BAI701- Deep Learning- Module 5 Notes

 

Module 5

Deep Reinforcement Learning: Introduction, Stateless Algorithms: Multi-Armed Bandits, The Basic Framework of Reinforcement Learning, case studies.

Textbook – 3: Chapter 9: 9.1,9.2,9.3, 9.7

Deep Reinforcement Learning

9.1 Introduction

How Humans Learn vs. How Machines Learn

🔹 Human Learning:

• We, humans, don’t learn by looking at fixed training examples like computers do.
• Instead, we learn continuously through experiences.
• We make decisions, and based on whether we get a reward (something good) or punishment (something bad), we change our behavior for the future.
• This type of learning is called trial and error, guided by rewards.

For example: A child learns not to touch fire after getting burned once. This is learning from experience.

• Some of our instincts or intelligence are built into our genes, thanks to evolution – a natural trial-and-error process over millions of years.
• So, most of what we call intelligence in living beings is the result of interacting with the environment and learning from it.

Herbert Simon’s Ant Hypothesis:

"Humans seem complex because they live in a complex environment."

• Herbert Simon, a famous AI researcher, suggested that human behavior looks complex not because we are complex, but because the world around us is.
• Deep down, humans are simple: we are reward-driven and act based on what benefits us.

Reinforcement Learning (RL): Mimicking Human Learning

• Reinforcement Learning is a field in machine learning that tries to copy how humans learn:
• By trying things out
• Making mistakes
• Learning from rewards

 What is it?

• A method where an AI system learns from the environment by taking actions and receiving rewards or punishments.
• The aim is to maximize the total reward over time.

Example: Teaching a robot to walk by letting it try different ways and rewarding it when it moves forward.

Real-Life Applications of Deep Reinforcement Learning

1. Playing Video Games

• AI can learn to play games like those on the Atari 2600 console.
• It doesn’t know the rules; it just sees the pixels on the screen and tries random moves.
• With time and many mistakes, it gets better because it learns from rewards (game scores).
• In many games, AI has performed better than humans!

Why games? Because they are small, controlled worlds that resemble real-life decision-making situations.

2. Playing the Game of Go – AlphaGo

• Go is a complex board game, even harder than chess.
• Google DeepMind created AlphaGo, an AI that:
• Learned by playing against humans and itself
• Discovered new, creative strategies
• Beat the best human players
• It also learned chess and beat top engines.

3. Self-Driving Cars

• Self-driving cars use data from sensors to make driving decisions.
• Although many systems use supervised learning, reinforcement learning is also helpful.
• The car learns from feedback and can make fewer mistakes than humans over time.

4. Teaching Robots to Walk

• Reinforcement learning is also used in robot locomotion.
• Instead of showing how to walk, we just reward the robot for reaching a target.
• The robot tries crawling, rolling, etc., until it learns to walk on its own.
• This kind of learning makes robots more independent.

Why is Reinforcement Learning Useful?

• It works well in situations where:
• It's easy to say whether the result was good or bad, but
• It's hard to define the exact correct action.

Example: In a game like chess, we can say who won easily, but it’s hard to decide the best move at every turn.

• Reinforcement learning makes it possible to learn complex behaviors by just giving rewards and letting the AI figure things out.
• The complexity of behavior comes naturally from interacting with a complex environment.

Example: The Multi-Armed Bandit Problem

Imagine you're in a casino with many slot machines, but you don’t know which one gives the highest reward.

• Some machines give better payouts, but you don’t know which ones.
• You try different machines to learn about their rewards. This is called exploration.
• Once you find the best one, you want to play it more to get higher rewards. This is called exploitation.
• The challenge is to balance between trying new options and using what you already know.

This simple idea is the basis of many reinforcement learning problems.

Final Thought: Why It Matters

• Reinforcement learning is like teaching machines to learn by experience, just like we do.
• It is useful for building:
• Game AIs
• Self-driving cars
• Smart robots
• Any system that needs to make decisions in complex, changing environments
• It shows that AI doesn't always need to be told what to do, it can learn by trial and error, as long as it knows what outcome is desirable.

9.2 Stateless Algorithms: Multi-Armed Bandits

 The Gambler and Slot Machines: The Multi-Armed Bandit Problem

Imagine a gambler in a casino. There are several slot machines (let's say 5) in front of him. Each machine gives different amounts of rewards, but the gambler doesn’t know which one is best. So, he has to try different machines to learn which one gives the most money over time.

What’s the Problem?

The gambler faces a big challenge:

• If he keeps trying new machines, he’s exploring (this gives him more information).
• If he keeps playing the machine that seems best so far, he’s exploiting his knowledge.

But here’s the trick:

• If he explores too much, he wastes time on machines that might not be good.
• If he exploits too soon, he might miss a better machine just because he didn’t explore enough.

This is called the exploration vs. exploitation dilemma.

Why This Matters in Learning?

This is a basic idea in reinforcement learning.

• In video games, self-driving cars, or robotics, the system has to learn by trying different actions.
• Some actions may look good but may not work in the long run.
• Others may seem bad at first but turn out to be better after more experience.

So, algorithms need a smart way to balance exploring new things and using what they already know.

 A Key Difference: No "State"

In the slot machine case:

• The machine gives the same kind of reward every time for a given arm.
• So, there’s no change in the environment or “state.”

But in real-world cases like:

• A video game (where the scene changes),
• A robot walking (where its position changes), or
• A car driving (where the road conditions change),

…the situation depends on the “state”—that is, what’s happening around the system at the time.

So, the multi-armed bandit is a simplified version of reinforcement learning, useful for understanding the basic idea before going into more complex state-based scenarios.

Strategies for Balancing Exploration and Exploitation

There are smart strategies to help the gambler choose when to explore and when to exploit:

1. ε-Greedy Strategy:
• With probability ε (like 10%), try a random machine (explore).
• With probability 1 - ε (like 90%), pick the best-known machine (exploit).
2. Upper Confidence Bound (UCB):
• Choose the machine not just based on reward, but also on how uncertain we are about it.
• This allows trying machines we’re less sure about.
3. Thompson Sampling:
• A probabilistic method where the algorithm chooses machines based on both past rewards and chances of being better.

These methods help balance learning and earning — and are useful not only in games but also in business, robotics, and AI.

Summary for Students

• Multi-armed bandit is a basic example used to explain how AI learns by trial and error.
• It teaches us how to make good decisions over time by balancing what we know vs. what we need to explore.
• This concept is the foundation of reinforcement learning.
• Even though it’s simple (no state), it helps us understand more complex learning environments like video games, robots, and smart cars.

Would you like a diagram or flowchart to illustrate this further for your students?

9.2.1 Naive Algorithm

Fixed Exploration Followed by Exploitation: A Simple but Risky Strategy

Let’s continue with the slot machine example.

One simple idea the gambler can try is this:

• Step 1 (Exploration Phase): Try each machine for a fixed number of times — like 10 times each — just to see how much reward each gives.
• Step 2 (Exploitation Phase): After testing all machines, pick the one that gave the most rewards and use only that machine forever.

At first, this method may seem smart and simple — try everything, then stick with the best.

But There Are Some Big Problems

1.  How Many Times Should You Try Each Machine?

• It’s very hard to decide how many times you should test each machine.
• What if you don’t try enough and miss a good machine just because it didn’t give a reward during your limited trials?
• Or what if you try too many times and waste time on bad machines?

2. Slow to Learn Rare Rewards

• Some machines might give rewards only once in a while (rarely).
• If you stop exploring too soon, you might not discover that those machines actually give higher rewards in the long run.

3. Choosing the Wrong Machine Forever

• If you make a wrong decision after the exploration phase, you’re stuck.
• The strategy says to never change again, so you might keep using the wrong machine forever — and lose out in the long run.

Why This Strategy Fails in Real Life

In the real world, situations are not fixed:

• New opportunities can come up.
• The environment can change.
• We learn continuously, not just in two separate steps.

So, this “explore-then-stick-forever” approach doesn’t work well in practice. It’s too rigid and doesn’t allow adapting to new information.

What’s Better?

A better approach is to keep exploring a little, even after you’ve found a good option. This way:

• You stay open to better options that may appear.
• You correct mistakes if you initially picked the wrong machine.
• You adapt as things change.

Summary for Students

• The fixed strategy (explore then exploit forever) is too simple for real-life problems.
• It’s hard to decide how much to explore.
• It can lead to wasting time or making wrong choices.
• Real-world learning should be ongoing, flexible, and adaptable.

9.2.2-Greedy Algorithm

ε-Greedy Algorithm: A Smart Balance Between Exploration and Exploitation

Let’s go back to our gambler and slot machine example.

The ε-greedy algorithm (pronounced “epsilon-greedy”) is a better strategy than “explore first, exploit forever.” It helps the gambler learn smartly and use the best machine more often, without completely ignoring other machines.

How Does ε-Greedy Work?

This method divides each trial (each play of a slot machine) into two types:

1. Exploration (with probability ε)

• With a small chance (ε), the gambler chooses a random machine, even if it’s not the best so far.
• This allows gathering new information that might reveal a better machine later.

2. Exploitation (with probability 1−ε)

• Most of the time (1−ε), the gambler chooses the machine that has given the highest average reward so far.
• This ensures that he benefits from what he has already learned.

Example:
If ε = 0.1 (or 10%), then:

• 10% of the time, the gambler will try a random machine.
• 90% of the time, he will use the best machine found so far.

 Why ε-Greedy Is Better

1. Avoids Being Stuck with the Wrong Machine
• Even if the gambler made a wrong guess early on, exploration helps him discover better machines later.
2. Starts Using Good Machines Early
• Unlike fixed-exploration methods, this strategy starts exploiting early, so the gambler gets better rewards sooner.
3. Learns While Earning
• The gambler keeps learning even during the main phase, which is much more realistic and effective.

But What’s the Right Value of ε?

• ε is a setting that the algorithm designer must choose.
• A typical value might be ε = 0.1, but:
• If ε is too small, the gambler might miss out on better machines.
• If ε is too large, the gambler wastes time exploring too much.

Annealing ε: Smarter Over Time

One smart trick is to change ε over time, called annealing:

• Start with a large ε (more exploration) in the beginning.
• Slowly reduce ε over time (less exploration, more exploitation).
• This means: explore a lot when you know nothing, and explore less once you’ve learned enough.

Summary for Students

• The ε-greedy algorithm is a clever way to balance learning and earning.
• It sometimes explores to keep improving.
• It mostly exploits to get the best reward.
• Choosing ε wisely (or decreasing it over time) makes the strategy flexible and effective.
• It’s commonly used in reinforcement learning systems, including games, robotics, and AI decision-making.

9.2.3 Upper Bounding Methods

Upper Confidence Bound (UCB): A Smarter Way to Learn in Slot Machine Problems

We’ve already learned about the ε-greedy method, where the gambler sometimes explores and sometimes exploits. But now, let’s look at a more intelligent method called the Upper Confidence Bound (UCB) strategy.

This method helps the gambler explore and exploit at the same time, and in a more efficient way!

What’s the Main Idea Behind UCB?

Instead of just using the average reward of a slot machine (like in ε-greedy), the gambler in the UCB strategy considers how uncertain he is about each machine’s performance.

He calculates a score for each slot machine, called the Upper Confidence Bound (Ui):

Ui=Qi+CiU_i = Q_i + C_i

• QiQ_i: The average (expected) reward of machine i so far (like a report card).
• CiC_i: A bonus value that depends on how uncertain he is about the machine.

This bonus encourages the gambler to try machines he hasn't tested much yet—just in case they turn out to be great!

Why Is UCB Better?

1. No Separate Phases:
• Unlike ε-greedy, we don’t divide trials into “explore” or “exploit.”
• Every trial is a smart mix of both — based on uncertainty and past performance.
2. Gives New Machines a Fair Chance:
• Machines that haven’t been tested much have high CiC_i, so they’re tried more.
• If they’re good, they’ll be used more in future.
3. Avoids Wasting Too Many Trials:
• It doesn’t keep trying bad machines for long.
• As soon as it finds that a machine isn’t good, it quickly lowers its score.
4. Fully Automated Balance:
• The choice of which machine to try is automatically balanced based on stats.

Choosing the Right K

• KK decides how much bonus we give for uncertainty.
• A higher KK means we explore more (more careful).
• A lower KK means we explore less and exploit more quickly.
• For example, K=3K = 3 gives us a 99.99% confidence level if we assume rewards follow a bell-curve (Gaussian).

Summary for Students

• The UCB method helps a gambler act smartly at every trial — choosing the best-looking machine while also trying out uncertain ones.
• It avoids dividing trials into separate explore and exploit parts — it does both together using a confidence-based score.
• It is more efficient than ε-greedy, especially when the number of trials is limited.
• UCB is widely used in multi-armed bandits, and its logic forms the foundation of more advanced reinforcement learning methods.

9.3 The Basic Framework of Reinforcement Learning

Understanding Reinforcement Learning (RL) in Simple Words

In basic bandit algorithms (like slot machine problems), each decision happens in the same situation over and over again. The environment doesn’t change. But in real-world problems (like playing video games or controlling a self-driving car), the situation or condition—called the state—keeps changing. This is where Reinforcement Learning (RL) is needed.

 Real-World Example: Self-Driving Car or Video Game

In real life:

• Every action you take changes the current condition (state).
• Your reward depends not just on one move but on the sequence of previous moves.

For example:

• In a video game, just pressing a button once might not earn you a point. But the reward may come after many correct moves.
• In a self-driving car, turning sharply may be bad in a normal situation, but necessary in an emergency. The value of that action depends on the current situation (state).

AGENT - ENVIRONMENT INTERACTION

Reinforcement Learning is all about learning by interacting with the environment. It works like this:

1. Agent (e.g., player or self-driving car) takes an action.
2. The environment responds by giving:
• A reward (e.g., points in a game or reaching the right destination).
• A new state (e.g., new position or condition).
3. The agent learns from this feedback and improves future decisions.

Example: A mouse in a maze:

• Action: Turns left.
• Reward: Finds cheese or not.
• New State: New position in the maze.
• Learning: Brain updates based on whether cheese was found.

Over Time:

The agent tries different actions in different situations. With experience, it learns:

• Which actions lead to good rewards.
• Which situations (states) need what kind of actions.

This is the core idea of reinforcement learning:

“Learn the best actions to take in different situations by interacting with the environment and using past rewards to improve.”

Markov Decision Process (MDP)

Reinforcement learning problems are modeled using Markov Decision Processes (MDPs).

🧩 What is an MDP?

An MDP consists of:

• States (S): All possible situations or positions.
• Actions (A): All possible moves or decisions the agent can make.
• Rewards (R): The feedback the agent gets after each action.
• Transition Rules: How one state moves to the next after an action.

Key Property of MDP:
The current state has all the information needed to decide what happens next—this is called the Markov property.

Episodic vs. Non-Episodic Tasks

1. Episodic Tasks:
• These have a clear end.
• Example: Tic-tac-toe or a level in a video game.
• The learning is based on a finite sequence:
• s0, a0, r0, s1, a1, r1, ..., sn, an, rn

where s is the state, a is the action, and r is the reward.

2. Non-Episodic Tasks:
• These go on forever.
• Example: A robot that works continuously without stopping.

Learning in Biological Systems

• In humans or animals, rewards are not just about the last action, but often depend on a whole chain of actions.
• When a mouse learns to find cheese in a maze, its brain (neurons) strengthens the memory of the sequence of actions that led to the reward.
• Deep Reinforcement Learning uses neural networks to do this learning in computers, using data from the environment (like game visuals or sensor inputs).

Summary

• Reinforcement Learning is all about learning from actions and rewards in changing situations.
• The agent uses its experience to learn which actions are best in different states.
• This process is modeled using Markov Decision Processes (MDPs).
• Real-world problems like games, robotics, and self-driving cars need RL, because their environments change and depend on past decisions.

Examples

In reinforcement learning, the state refers to everything we need to know about the environment at a particular time. However, in real-world applications, we often don’t use the full environment details. Instead, we use a simpler version or an approximation of the state to make learning easier. Let’s look at a few examples to understand this better:

1. Tic-Tac-Toe, Chess, or Go (Board Games):

• State: The arrangement of the pieces on the board at any point in time.
• Action: The move made by the player (or agent).
• Reward: At the end of the game:
• +1 if the agent wins,
• 0 for a draw,
• −1 if the agent loses.
• Important Point: Sometimes, the agent might make a very smart move, but it doesn't get any reward right away. The reward only comes at the end of the game.

2. Robot Walking (Locomotion):

• State: The position of the robot and the angles of its joints (e.g., how its legs are placed).
• Action: Applying force or torque to different joints to make the robot move.
• Reward: Given at each step, based on:
• Whether the robot is still standing (not fallen),
• How much progress it made in moving forward.

3. Self-Driving Car:

• State: The car’s current situation, captured by sensors (e.g., cameras, lidar, GPS).
• Action: The car’s decisions—like turning the steering wheel, accelerating, or braking.
• Reward: A custom-designed formula that depends on:
• How far the car travels (progress),
• How safely it drives (avoiding collisions).

In all these examples, we first need to carefully choose:

• What we will treat as the state, and
• How we will define the reward.

Once we define these correctly, reinforcement learning can take over and build a complete system that learns automatically from experiences. This is called an end-to-end learning system, because it handles everything—from sensing the environment to choosing actions and improving over time—by itself.

9.3.1 Challenges of Reinforcement Learning

Reinforcement learning (RL) is more challenging than supervised learning (like training a model to recognize handwritten digits or spam emails) for several important reasons:

1. Credit Assignment Problem:

• Imagine you win a game of chess.
• You made many moves throughout the game, but you don’t know exactly which moves helped you win.
• This is called the credit-assignment problem—figuring out which actions (moves) were responsible for the final reward (winning).
• It gets even harder when rewards are random or uncertain, like pulling the lever of a slot machine, where you don’t always win even if you do the same action.

2. Huge Number of States:

• In many RL problems, there are millions or billions of different situations (called "states")—for example, every different way a chessboard can be arranged.
• The agent can’t experience all possible situations during training.
• So, it needs to generalize—to make smart decisions even in new situations it has never seen before.
• This is where deep learning helps: it allows the system to recognize patterns and generalize from past experiences.

3. Exploration vs. Exploitation:

• In RL, there’s a trade-off:
• Should the agent try new actions to discover better options (exploration)?
• Or should it stick to the known best actions to get good rewards (exploitation)?
• If it explores too much, it may waste time or get poor results.
• If it exploits too much, it might miss better options it hasn’t tried yet.
• Finding the right balance is a key challenge in RL.

4. Data Collection is Hard:

• In supervised learning, we can collect lots of labeled data from files or images.
• But in RL, the agent has to interact with the environment to learn, often by trying and failing.
• For example:
• A robot has to move around and learn from trial and error.
• A self-driving car must be tested on real roads—but this can be dangerous and expensive.
• Most of the time, in early learning stages, the agent may fail a lot before improving.
• That’s why RL is often tested in games or simulations, where failures are safe.
• But applying it in real-world physical systems is very hard and is currently the biggest challenge in reinforcement learning.

9.3.2 Simple Reinforcement Learning for Tic-Tac-Toe

We can improve the basic greedy algorithm (used in multi-armed bandits) to make it work for simple games like tic-tac-toe.

Understanding the Problem:

• In tic-tac-toe, each board position is considered a state.
• An action is placing an ‘X’ or an ‘O’ in one of the empty positions.
• There are 3 choices for each of the 9 squares on the board (‘X’, ‘O’, or empty), so the total number of possible board states is at most 3⁹ = 19683.

From Stateless to State-Action Learning:

• In multi-armed bandits, we estimated the value of each action (without considering any state).
• But now, in tic-tac-toe, we estimate the value of each action in each specific state—this is called the state-action pair (s, a).
• We estimate how good an action a is when taken in a state s, based on how it performed in the past (win/loss/draw).

Reward Calculation Using Discounting:

• We use a discount factor γ (gamma) where 0 < γ < 1 to give more importance to earlier wins.
• Suppose the game ends in a win after r moves, then the action that helped win gets a reward of γ^(r–1).
• If the game ends in a loss, we subtract that value: –γ^(r–1).
• If it’s a draw, the reward is 0.
• This means faster wins are better and delayed wins or actions have less impact (which mimics real-world situations where immediate results are more valuable).

How the Learning Happens:

• The algorithm stores these unnormalized reward values in a table.
• After the game is over, it updates the table using the final result.
• To get the average (normalized) value of each action, it divides the total reward by the number of times that action has been used in that state.

Choosing Actions (Exploration vs. Exploitation):

• At the beginning, the table is filled with small random values.
• During the game:
• With probability 1 − ε (epsilon), we choose the best action (with the highest average reward).
• With probability ε, we choose a random action—this helps in exploration.
• All moves are updated after the game ends (this is called offline updating).

Improving the Learning (Self-Play):

• Instead of playing against a fixed opponent, we can make the agent play against itself. This is called self-play.
• During self-play, the value of an action is updated as:
• +γ^r if the move leads to a win
• 0 for a draw
• −γ^r for a loss
• This helps the system learn optimal strategies on its own.

Making Moves After Training (Inference Time):

• Once training is done, the agent uses the table to pick the move with the highest average value for any given state.

9.3.3 Role of Deep Learning and a Straw-Man Algorithm

 

Understanding Traditional vs. Deep Reinforcement Learning

🔸 Traditional Reinforcement Learning and Tic-Tac-Toe

In the earlier example, we used a basic algorithm (ε-greedy method) to train a program to play tic-tac-toe. This method used a table that stored values for every move (called state-action pairs) to decide which move was good or bad based on outcomes from past games.

• Each game board was treated as a state.
• Every possible move (like placing an 'X' or 'O') was an action.
• The algorithm learned values for each state-action pair through trial-and-error (i.e., playing many games).
• Good moves that led to wins got positive values. Bad moves got negative values.
• Shorter wins were given more credit using a discount factor γ < 1, meaning winning in fewer moves was better.
• The table values started with small random numbers and were updated only after the game finished.

This worked well for tic-tac-toe because the number of possible board positions is small (less than 20,000). But what about complex games like chess?

Limitations of Table-Based Learning

In real games like chess:

• The number of possible board states is enormously large—far too many to store in a table.
• It’s impossible to see every possible game state through trial and error.
• So, you can’t just memorize all winning and losing situations like you could with tic-tac-toe.

Therefore, using a big table to store values for each move in every situation is not practical for big problems like chess, Go, or real-world robotics.

The Need for Generalization

Humans learn not just by memorizing exact situations, but by understanding patterns. For example:

• If a chess player has seen a type of trap before, they can recognize a similar one even if the exact pieces are in different spots.
• This is called generalizing knowledge — learning something in one situation and applying it to a new but similar situation.

Machines can do this too—but only with the help of deep learning.

What Does Deep Learning Do in Reinforcement Learning?

Deep learning replaces the big table of state-action values with a neural network:

• The input is the current state (for example, the current game board).
• The output is the value of making a particular move.
• Instead of storing values for each move in a table, the neural network learns a function to predict the value of moves.
• This is trained by feeding many examples (from past games), and using backpropagation to improve its predictions.

So even if the machine sees a completely new board position, the neural network can still predict a good move based on what it learned earlier.

Example: AlphaZero vs. Stockfish

AlphaZero is a deep reinforcement learning system created by Google DeepMind. It learned to play chess, not by studying human games, but by playing against itself thousands of times (self-play).

Some key points:

• It learned to make strategic sacrifices, like giving up material (pieces) to trap the opponent or control the board.
• These strategies are hard to program manually, but AlphaZero learned them through experience.
• It learned positional understanding, not just based on material advantage (how many pieces), but how well the pieces work together—just like expert human players.
• AlphaZero’s neural network evaluated board positions holistically, meaning it could see the big picture better than traditional programs like Stockfish that rely on human-crafted rules.

Real-Life Applications

In real life, we don’t always know the exact "state" of the environment in neat table form. For example:

• In robotics, the input might be camera images or sensor data.
• In games, the input might be the raw pixels of the game screen.

Deep neural networks can take these raw inputs, find the important patterns, and decide what action to take.

Just like humans don’t have a lookup table for every possible situation, deep learning allows machines to make smart decisions in new situations, using knowledge learned from past experience.

Key Takeaways

Concept	Traditional RL (like in Tic-Tac-Toe)	Deep RL (like AlphaZero)
Memory	Table of state-action pairs	Neural network model
Scalability	Only works for small problems	Works for large, complex problems
Generalization	Poor (memorizes only seen states)	Good (can handle new states)
Input	Clean game states	Raw sensory data or pixels
Update	After full game	After every move (or mini-batch)
Learning	From limited samples	From massive self-play or simulation

9.7 Case Studies

In the following, we present case studies from real domains to showcase different reinforcement learning settings. We will present examples of reinforcement learning in Go, robotics, conversational systems, self-driving cars, and neural-network hyperparameter learning.

9.7.1 AlphaGo: Championship Level Play at Go

Understanding the Complexity of Go and Deep RL

🔹 What is Go?

Go is a two-player board game, just like chess or tic-tac-toe, but it's much more complex.

• The board has 19×19 lines, creating 361 intersections (where you place stones).
• Two players take turns placing stones — Black goes first with 181 stones, and White has 180 stones.
• Stones are not moved once placed.
• You can capture the opponent’s stones by surrounding them completely.
• The goal is to control more area on the board than your opponent.

🔹 How is Go different from Chess or Tic-Tac-Toe?

Let’s compare:

Game	Board Size	Avg. Number of Moves (Branching Factor)	Average Game Length (Tree Depth)
Tic-Tac-Toe	3×3	Very small (~2–5)	Very short
Chess	8×8	About 35 moves possible per turn	~70–80 moves total
Go	19×19	About 250 moves per turn	~150 moves total

 That means:

• Go has 10× more possible moves per turn than chess.
• Go games are usually twice as long as chess games.
• If you try to create a tree of all possible Go games (minimax tree), it becomes astronomically large—even bigger than the number of atoms in the universe!

🔹 Why Traditional Chess Algorithms Fail at Go

Traditional chess programs use a strategy called minimax tree search:

1. They simulate many possible moves and countermoves (up to a certain depth).
2. They evaluate the board at the end of each simulation using hand-crafted rules (like counting remaining pieces).
3. They prune (cut off) moves that are clearly worse to save time.

This works in chess because:

• There are rules and values for pieces.
• Fewer moves make it easier to simulate many possibilities.

But this strategy does not work well for Go:

• The board is much bigger.
• There are no "piece values" like in chess.
• Too many possible moves.
• It needs spatial intuition and pattern recognition, which are very hard to program manually.

How Do Humans Play Go?

Humans:

• Don't calculate all moves.
• Learn visual patterns from experience.
• Use intuition to play good moves in complex situations.

This is exactly what Reinforcement Learning (RL) tries to imitate.

How AlphaGo Plays Go

AlphaGo is a deep reinforcement learning system that learns like a human, using:

1. Convolutional Neural Networks (CNNs)

• The Go board is treated like an image of 19×19 pixels.
• Each intersection (where stones can be placed) is encoded as binary features.
• Multiple feature maps are used:
• One map shows which positions are occupied.
• Another shows how many turns have passed since a stone was placed.
• Others track the status of each intersection.
• Altogether, AlphaGo uses 48 binary planes, each of size 19×19.
• These go into a deep CNN that can recognize spatial patterns, just like how CNNs see features in images.

2. Policy Network

• This network learns which move to make in a given board state.
• It’s trained on:
• Thousands of expert Go games.
• Games AlphaGo plays against itself (self-play).
• Over time, it learns policies — strategies for choosing strong moves.

3. Value Network

• Instead of waiting until the end of the game to see who wins, this network predicts the expected outcome from a current board state.
• It answers: “Is this a good or bad position to be in?”

4. Monte Carlo Tree Search (MCTS)

• This is used during actual play.
• It simulates many possible games from the current position.
• Combines predictions from the policy and value networks to decide the best move.

Why AlphaGo is Special

AlphaGo does not rely on brute force. Instead:

• It learns from experience.
• It generalizes knowledge to new board positions.
• It uses deep neural networks to evaluate and play like a human expert.

Because of this, AlphaGo defeated world champions—a milestone in both AI and the history of the game of Go.

Summary Table

Component	Description
CNN Input	48 binary feature maps of 19×19 board state
Policy Network	Predicts good moves from the current board
Value Network	Predicts the outcome (win/loss probability) from the current state
Monte Carlo Tree Search	Simulates possible future games to choose best moves
Training	From expert games + self-play using reinforcement learning
Output	Plays Go at a superhuman level by imitating patterns and strategy

Policy Networks

What is a Policy Network?

A policy network is a neural network that takes the current Go board as input and tells us how likely each possible move is.

• The input is the visual map of the board (with 19×19 positions, and extra features like how many moves ago a stone was placed).
• The output is a probability for each possible move (i.e., placing a stone on each empty position on the board).
• The activation function used at the output layer is softmax, which turns the values into probabilities that all add up to 1.

Structure of the Policy Network

• It has 13 convolutional layers with ReLU activation (which adds non-linearity).
• Most layers use 3×3 filters, except:
• The first layer uses a 5×5 filter.
• The last layer uses a 1×1 filter.
• Zero padding is used to keep the size of the image the same throughout.
• It uses 192 filters in each layer.
• No max pooling is used — this means the spatial layout (positions of stones) is preserved at every layer.

How is the Policy Network Trained?

AlphaGo used two different training methods for two policy networks:

1. Supervised Learning (SL-policy network)

• This method is like learning from a teacher.
• Input: The board positions from real games played by expert Go players.
• Output: The actual move made by the expert in that situation.
• This is also called imitation learning, because the network is learning to imitate experts.
• Every move by an expert is assumed to be a good move (+1 reward).
• The network is trained using backpropagation with the goal of increasing the probability of the expert’s move.

Why do this?

• It gives the network a good starting point, especially when it doesn’t know anything (avoids the "cold-start" problem).
• But later, researchers found that skipping this step and directly using reinforcement learning could also work well.

2. Reinforcement Learning (RL-policy network)

• In this method, the network learns by playing games against itself.
• Each version of the network plays against older versions to make the training more diverse.
• After each full game:
• If it wins, it gets a +1 reward.
• If it loses, it gets a −1 reward.
• Every move made in the game is labeled with the final game result.
• This data is used to train the network so that moves leading to wins become more likely in the future.

Why do this?

• This method allows the network to improve by trial and error, even beyond human experts.

Outcome

Both SL and RL policy networks became very strong Go players, even better than most existing software at that time.

These networks were later combined with Monte Carlo Tree Search (MCTS) — a method to explore many future move possibilities — to further boost the performance.

Key Takeaways for Students

• A policy network helps decide what move to make by predicting probabilities for each legal move.
• Supervised learning teaches the network by copying experts.
• Reinforcement learning helps it learn from its own experience.
• The network gets better with more games and more training — just like a human who practices a lot!

Value Networks

 What is a Value Network?

The value network is a type of convolutional neural network (CNN) that takes the current state of a Go game as input and predicts a score between −1 and +1.

• +1 means the current player (who is about to move) is very likely to win.
• −1 means the current player is very likely to lose.
• A value near 0 means the game is balanced or uncertain.

So, the value network doesn't tell you what move to make — it tells you how good a particular board position is for the player who is about to move.

Input and Output of the Value Network

• Input: The visual representation of the Go board (just like the policy network), plus an extra feature that indicates whether the current player is White or Black.
• Instead of calling the colors "white" and "black", the network uses the idea of "current player" and "opponent" to make things more general.
• Output: A single number between −1 and +1.
• This is done using a tanh activation function at the final layer.

Architecture of the Value Network

• The early layers (convolutional layers) are almost the same as the policy network.
• But there are a few key differences:
• One extra convolutional layer is added around layer 12.
• After the final convolutional layer, there's a fully connected layer with 256 neurons and a ReLU activation.
• The last layer has just one output neuron using tanh, which gives the final score between −1 and +1.

How is the Value Network Trained?

To train the network, we need lots of examples of:

• Game positions (the board state), and
• Outcomes (did the player win or lose from that position?)

There are two ways to get these examples:

Option 1: Use real Go games

• One option is to use positions from professional or amateur Go games.
• But these games don’t always go to the end or may not give enough variety.

Preferred Option: Self-play

• The better approach was to use self-play using the already trained SL-policy and RL-policy networks.
• The network plays complete games against itself and generates thousands of positions along with final outcomes (win/loss).
• These (position, outcome) pairs are then used to train the value network.

Important Note: Preventing Overfitting

• In one game, the board positions are highly related (they come one after another).
• If we use too many similar positions from the same game, the network may memorize instead of learning to generalize — this is called overfitting.
• To avoid this, the training examples are chosen from different games, so that each example is more independent.

Key Takeaways for Students

• The value network learns to predict the chance of winning from any board position.
• It’s trained using data from self-play games.
• Its architecture is similar to the policy network, but the output is a single score instead of a set of probabilities.
• It helps AlphaGo evaluate board positions more intelligently, rather than just relying on simulations.

Monte Carlo Tree Search

What is Monte Carlo Tree Search (MCTS)?

Monte Carlo Tree Search is a smart way of exploring possible moves in a game (like Go) to decide what the best next move should be.

• Think of it as a look-ahead strategy: it builds a tree of future possible moves.
• Each node in the tree is a possible state of the game.
• The goal is to explore the tree to find the most promising next move.

What Makes AlphaGo’s MCTS Special?

AlphaGo uses a combination of techniques to make its search faster and smarter. Let's break it down:

Two Ways to Evaluate a Leaf Node

When MCTS reaches a leaf node (a game state that hasn’t been explored before), it needs to estimate how good that position is. AlphaGo uses two evaluations:

1. Rollout Evaluation (e₁) — Based on Fast Simulations

• AlphaGo simulates quick games from that position to the end and sees whether it wins or loses.
• This is called a rollout.
• Instead of using the full neural network (which is slow), AlphaGo trained a simplified softmax classifier using:
• Human Go game data
• Some hand-crafted features (rules or patterns made by experts)
• This made the simulations much faster, even though they were a bit less accurate.

2. Value Network Evaluation (e₂) — Based on Neural Network

• AlphaGo also uses the value network to directly predict how likely it is to win from that position.
• This is more accurate but slower than rollouts.

Combining the Two Evaluations

To make the final decision, AlphaGo combines both evaluations using a formula:

e=β⋅e1+(1−β)⋅e2e = \beta \cdot e_1 + (1 - \beta) \cdot e_2

• e₁ = result of the fast simulation (rollout)
• e₂ = result of the value network
• β is a number between 0 and 1 that balances how much weight we give to each method.

In AlphaGo:

• β was set to 0.5, meaning equal weight was given to both evaluations.
• But even using only the value network worked well — it was almost as good and simpler!

How MCTS Chooses the Best Move

• After many simulations (thousands!), MCTS keeps track of how often each move was chosen during the simulations.
• The most visited move is selected as the final predicted move — the one AlphaGo plays.

What’s K = 1/√(N(s, b))?

This part refers to a mathematical trick used during exploration:

• It helps the algorithm explore less-visited paths more.
• N(s, b) = number of times action b was taken from state s.
• K = 1 / √(N(s, b)) means the less you’ve explored a move, the more you’re encouraged to try it.

This improves exploration, so the system doesn't get stuck trying only a few familiar moves.

Summary for Students

Concept	Simple Explanation
MCTS	Explores possible moves using a tree to find the best one.
Rollout (e₁)	Fast simulation using a simplified model to guess how good a move is.
Value Network (e₂)	Accurate prediction using a deep neural network.
Final Score (e)	A combination of e₁ and e₂ (usually 50–50) to estimate how good a move is.
Most Visited Move	After all simulations, the move visited most is selected as the best.
Exploration Term	Encourages trying less-explored moves to avoid missing better paths.

9.7.1.1 Alpha Zero: Enhancements to Zero Human Knowledge

Understanding AlphaGo Zero and AlphaZero – In Simple Terms

What was AlphaGo before?

Earlier versions of AlphaGo (the AI that beat human champions at the game of Go) used two separate neural networks:

• Policy Network: Suggested the next best move.
• Value Network: Estimated how good the current board state is (i.e., chances of winning from there).

It was also trained using human expert games — meaning, it learned from how humans played.

What Changed in AlphaGo Zero?

AlphaGo Zero improved on AlphaGo by removing the need for human knowledge completely. It didn’t learn from human moves anymore.

Instead, it used:

• A single neural network that gave:
• p(s, a) → Probabilities of taking each action a in a given state s (this is the policy).
• v(s) → Value of that board state (i.e., how good the situation is).

This was a major change! Now, the AI could learn completely by playing against itself (called self-play), without any human teaching.

Loss Function – How It Learned

To improve the neural network, a combined loss function was used:

• [v(s) − z(s)]²: The squared difference between predicted value v(s) and actual outcome z(s) of the game. This is the value loss.
• −π(s, a) log[p(s, a)]: Cross-entropy loss between actual improved action probabilities π(s, a) and predicted ones p(s, a). This is the policy loss.
• + λ||W||²: A regularization term to keep the weights (W) of the neural network from getting too large.

Role of Monte Carlo Tree Search (MCTS)

AlphaGo Zero used Monte Carlo Tree Search (MCTS) to make better decisions.

Here’s how:

1. At each state s, it runs multiple simulations using the tree search.
2. These simulations guide it to explore the best possible moves (actions a).
3. From this, it gets:
• Visit count N(s, a): How many times each move was visited during the search.
• Improved policy π(s, a): Based on these visit counts. This works like a better version of the original policy p(s, a).
• Q(s, a): Average value of taking action a in state s.

How Training Works (Self-Play and Bootstrapping)

1. Self-play: The AI plays full games against itself, using MCTS to decide each move.
2. For each move:
• It records the game state s
• The improved policy π(s, a)
• And the final game outcome z(s) (like +1 for a win, -1 for a loss).
3. These are used to create training data:
• Input: board state s
• Targets:
• π(s, a): better move probabilities (from MCTS)
• z(s): actual outcome of the game (win/loss)
4. This data updates the network, improving both its policy and value predictions.

This process keeps repeating — the improved network plays more games, generates better data, and learns from it.

Key Insight: Bootstrapping

Just like temporal-difference (TD) learning in RL estimates the value of a state using future estimates, here we use:

• Visit counts from MCTS to improve the policy
• Simulation results to improve the value estimate

This is called bootstrapping – using our current knowledge to improve future predictions.

What is AlphaZero?

AlphaZero is the next step after AlphaGo Zero. It’s a more general version that can learn:

• Chess
• Shogi (Japanese chess)
• Go

And it does this using the same algorithm (self-play, MCTS, combined network for policy and value).

AlphaZero's Achievements

• Beat Stockfish (the world’s strongest chess engine)
• Beat Elmo (top shogi program)
• This was surprising because many experts thought chess needed hand-coded strategies, but AlphaZero learned everything from scratch just by playing games with itself!

Summary: What Makes AlphaGo Zero and AlphaZero Special?

Feature	AlphaGo	AlphaGo Zero	AlphaZero
Training data	Human games + self-play	Self-play only	Self-play only
Networks	Two (policy + value)	One combined network	One combined network
Search	MCTS for inference	MCTS for training + inference	MCTS for training + inference
Games played	Go	Go	Go, Chess, Shogi
Human knowledge	Required	Not required	Not required

Comments on Performance

AlphaGo and AlphaZero: A Closer Look at Their Extraordinary Performance

AlphaGo’s Success Against Computers and Humans

AlphaGo, the AI program developed by DeepMind to play the game of Go, has shown amazing performance:

• It played 495 games against other Go software and won 494 of them!
• Even when it started with a disadvantage (by giving its opponent 4 extra stones), it still performed extremely well:
• It won 77% of games against Crazy Stone.
• It won 86% against Zen.
• It won 99% against Pachi.

That’s not all — it also defeated top human players, including:

• The European Champion
• The World Champion
• The highest-ranked player in the world

What Made AlphaGo’s Wins So Special?

AlphaGo didn’t just win — it won in creative and surprising ways.

• In many games, it played unusual moves that seemed wrong or strange at first.
• But after it won, experts realized those moves were actually brilliant and insightful.
• These moves were not traditional — they went against what human experts believed was “correct.”

This showed that AlphaGo had discovered new ways of thinking about the game — it had learned unique strategies on its own through self-play (playing against itself again and again).

After watching AlphaGo, even the best human Go players changed their understanding of the game.

AlphaZero’s Surprising Skill in Chess

AlphaZero, which came after AlphaGo Zero, showed the same kind of intelligence in chess.

• It did things that human grandmasters do, like:
• Giving up pieces (material sacrifice) to slowly gain better board control.
• Restricting the opponent’s movement and improving its own position step by step.

But what’s more surprising is:

• It didn’t follow traditional chess rules, like always keeping the king safe or valuing pieces in the usual way.
• It had no hard-coded rules or tips — it learned everything by playing games with itself.
• It even discovered famous chess openings that took humans hundreds of years to develop!
• And then, it formed its own opinions on which openings were actually better.

Learning by Itself – Reinforcement Learning in Action

This kind of smart, creative behavior is possible because of reinforcement learning (RL).

How is RL different from supervised learning?

• Supervised learning learns from already known answers (like learning from a teacher’s key).
• Reinforcement learning learns through trial and error:
• It tries things,
• Sees what works (gets a reward),
• Learns from experience.

So, AlphaGo and AlphaZero were not just copying existing strategies — they were:

• Discovering new ideas
• Learning from their own games
• Finding better ways to win

 Why This Matters for the Future

This ability to learn without being told what to do — just by getting feedback (rewards or penalties) — is a powerful feature of reinforcement learning.

It shows great potential in:

• Robotics
• Medical decision-making
• Autonomous vehicles
• Business strategy
• And many other real-world applications.

Because these AI systems aren’t limited to human knowledge, they can:

• Create their own strategies
• Adapt to new situations
• Invent new ways of solving problems

In Summary:

Concept	What Happened
AlphaGo's performance	Beat almost all computer and human players
Unique playing style	Played strange but brilliant moves
Impact on humans	Changed how top players understand the game
AlphaZero in chess	Sacrificed pieces like a human, found its own way to play
Key ability	Learned everything from self-play with no human input
Main technique	Reinforcement Learning – learning by doing, not by memorizing answers
Why it matters	Shows AI can go beyond human knowledge and innovate

9.7.2 Self-Learning Robots

Self-Learning Robots: An Exciting Area in Artificial Intelligence

What Are Self-Learning Robots?

Self-learning robots are robots that learn how to perform tasks by themselves — without being programmed for every specific situation. These tasks might include:

• Walking (locomotion)
• Fixing mechanical parts
• Picking up or retrieving objects

They use a method called reinforcement learning, which means:

• The robot tries different actions
• It gets rewards when it does something right
• Over time, it learns the best way to achieve its goal

Teaching a Robot to Walk

Let’s take a real-world example to understand how this works:

Suppose we build a robot that looks like a human — it has legs and joints and is capable of walking, at least in theory. However, it doesn’t yet know how to walk.
Just like a baby learning to walk, this robot has to figure out:

• How to move each joint
• How to stay balanced
• How to go from point A to point B without falling

Even though we humans walk naturally without thinking, it’s extremely difficult for robots, especially if they walk on two legs (bipedal robots).

Now imagine the robot has to walk on:

• Slippery floors
• Uneven ground
• Places with obstacles

This makes the challenge even harder.

Why Reinforcement Learning is Perfect Here

This walking problem is perfect for reinforcement learning because:

• It’s easy to judge if the robot is doing well (is it walking and reaching the goal?).
• But it’s hard to give exact instructions for every possible situation the robot might face.

So instead of programming it with step-by-step rules, we let the robot learn through experience:

1. The robot tries out random movements.
2. If it gets closer to the goal (point B), it gets a virtual reward.
3. If it falls or moves away from the goal, it gets no reward or a penalty.
4. Over time, the robot learns which movements help it stay balanced and walk properly.

Importantly:

• The robot isn’t given any example of how to walk.
• It’s not told what the “correct” actions are.
• It only knows that it will be rewarded for moving toward the goal.

So it slowly learns the right sequence of movements — just like how humans learn through trial and error.

Other Applications: Robot Manipulation

Reinforcement learning isn’t just useful for walking. It can also help robots learn how to:

• Grasp objects (like picking up a cup)
• Twist or turn things (like putting a cap on a bottle)

These are called manipulation tasks, and they are also hard to program directly because:

• The robot has to deal with different shapes and sizes of objects
• It needs to be gentle but precise
• The right way to do the task might change depending on the situation

Again, reinforcement learning is helpful because:

• The robot tries different ways to do the task
• It gets rewarded when it succeeds (e.g., screws the cap on)
• It learns by exploring and getting feedback

 In Summary

Task	Challenge	Why Reinforcement Learning Helps
Walking	Must balance and move correctly, especially on tricky terrain	Rewards help the robot learn the right movements through trial and error
Object manipulation	Needs to grasp or move objects in various ways	Robot can try different actions and learn what works best

Key Idea

Reinforcement learning allows robots to learn on their own, using a reward-based system. They don’t need to be told exactly what to do — they figure it out by trying, failing, and succeeding.

This kind of self-learning AI is very powerful and can be used in many areas where it's hard to write exact rules — like walking, repairing, helping people, or interacting with the environment.

9.7.2.1 Deep Learning of Locomotion Skills

Teaching Robots to Walk — Using Simulations Instead of Real Hardware

Training in a Virtual World

Instead of building real robots and testing them in the physical world (which can be expensive, slow, and sometimes dangerous), researchers first trained virtual robots using a powerful simulation tool called MuJoCo.

What is MuJoCo?

MuJoCo stands for Multi-Joint dynamics with Contact. It’s a physics engine, which means:

• It simulates real-world physical behavior (like movement, gravity, contact, friction, etc.).
• It is used in robotics, biomechanics, graphics, and animation.
• It helps researchers test robot movements safely without building an actual robot.

This makes it faster, cheaper, and safer to try out new robot designs and learning algorithms.

Types of Robots Used

Two types of simulated robots were trained in this research:

1. Humanoid robot (like a human – walks on two legs)
2. Quadruped robot (like a dog – walks on four legs)

Details of the models:

• Humanoid Robot:
• 33 state dimensions (to describe its position, joint angles, etc.)
• 10 actuated joints (joints that can be moved or controlled)
• Quadruped Robot:
• 29 state dimensions
• 8 actuated joints

 The Learning Objective

• The goal of both robots was to move forward — to walk.
• They were rewarded based on how much forward movement they made.
• The training would stop (terminate) if the robot fell down — i.e., when its center of mass dropped too low.

How Was the Robot Controlled?

• The robot’s actions were controlled by torques at its joints (just like muscles control human joints).
• The robot had access to sensors that provided information like:
• Position and angle of each joint
• Obstacle locations
• Orientation, etc.

This sensory data was given as input to a neural network, which helped decide what actions to take next.

Neural Networks Used

Two separate neural networks were used:

1. Policy Network: Chose the next action based on the current state (what to do).
2. Value Network: Estimated how good the current state is (how much reward to expect).

This setup is called an Actor-Critic method:

• The actor decides actions (policy).
• The critic evaluates how good the actions are (value).

 Structure of the Neural Network

Both networks used the same architecture for their hidden layers:

• 3 hidden layers with:
• 100 neurons (with tanh activation)
• 50 neurons (tanh)
• 25 neurons (tanh)

The only difference was in the output layer:

• Value network had 1 output (the estimated value).
• Policy network had multiple outputs — one for each possible action the robot could take.

Training the Model

To improve the learning:

• A technique called Generalized Advantage Estimation (GAE) was used.
• This helps calculate how much better a certain action is compared to average.
• Another method called Trust Region Policy Optimization (TRPO) was used.
• This ensures safe and stable updates to the policy during training.

After about 1000 iterations of training:

• The humanoid robot learned to walk with a smooth and natural gait (walking style).
• A video demonstration was shared showing the final walking behavior.

Realistic Learning, Real Results

Although this was done in simulation, the robot learned complex walking behaviors, which would otherwise be risky to try in the real world at first.
Later, Google DeepMind improved upon this and trained robots with even more advanced abilities, like:

• Avoiding obstacles
• Adapting to different terrains

Summary

Feature	Description
Simulation Tool	MuJoCo – a physics engine for realistic virtual robot training
Robot Types	Humanoid (2-legged) and Quadruped (4-legged)
Learning Method	Reinforcement Learning using rewards for forward motion
Network Architecture	3-layer feed-forward networks (100, 50, 25 tanh units)
Training Techniques	Actor-Critic, GAE, TRPO
Goal	Learn to walk without falling
Outcome	Successfully learned a natural walking motion in simulation

9.7.2.2 Deep Learning of Visuomotor Skills

Learning Household Tasks Using Reinforcement Learning

What Kind of Tasks Did the Robot Learn?

In this interesting example of reinforcement learning, a real physical robot was trained to do everyday household tasks, such as:

• Hanging a coat hanger on a rack
• Putting a block into a shape-sorting cube
• Using a toy hammer to fit its claw under a nail
• Screwing a cap onto a bottle

These are not easy tasks for a robot because each one requires precise movement, proper coordination, and correct force.

How Did the Robot Control Its Actions?

• The robot used 7-dimensional motor torque commands — this means it controlled seven joints at once (similar to a human controlling different parts of their arm).
• Each task required a sequence of movements — not just one simple action.

How Did the Robot “See”?

• A camera was used to give the robot visual input — this acted like the eyes of the robot.
• The robot processed camera images using a Convolutional Neural Network (CNN).
• This is similar to how our own visual brain system works — inspired by famous neuroscience experiments by Hubel and Wiesel.

Is It Like a Video Game?

Yes! Even though this setup may seem very different from games like Atari, there’s a big similarity:

• In Atari games, the AI also uses CNNs to understand raw images (game screens).
• Here, the robot also uses CNNs to understand camera images to decide what actions to take.

What Was the Neural Network Architecture?

The robot's neural network had 7 layers with about 92,000 parameters.

Here’s a breakdown:

Convolutional Layers (first 3 layers)

• Conv1: 64 filters, 7x7, stride 2, with ReLU activation
• Conv2: 32 filters, 5x5, ReLU
• Conv3: 32 filters, 5x5, ReLU

These layers extract useful features from the camera image.

Spatial Softmax Layer (4th layer)

• Instead of just selecting the strongest feature, this layer creates a probability distribution across the image.
• From this, it calculates the expected 2D position of important features in the image.
• This technique creates what’s called a “feature point”, which helps the robot understand where things are in the image.

This layer is unique and not usually found in standard CNNs — it’s specially designed to give a precise visual understanding for controlling movements.

Concatenation with Robot State

• After the CNN processes the image, the output is combined (concatenated) with additional robot state inputs, such as:
• Joint angles
• End-effector position (the robot’s hand)
• Velocities
• Object positions

The robot state had between 14 to 32 values, depending on the task.

These values were not given to the CNN directly because CNNs are better at understanding images, not robot states. So, they were added after the visual processing was done.

Fully Connected Layers (last 3 layers)

• Two hidden layers with 40 neurons (ReLU activation)
• Final layer: 7 output neurons corresponding to the 7 joint torques (actions)

What Was the Robot Learning to Do?

• The robot had to learn visuomotor skills — in other words, it needed to:
• See an object
• Decide what movement to make
• And control its arm with the right force and direction

It learned this by trial and error:

• If it succeeded (e.g., placed the hanger or screwed the cap), it got a reward.
• If not, it learned from the mistake and tried again.

 Supervised Learning Inside Reinforcement Learning?

A special method called Guided Policy Search was used.

• This technique takes some parts of the reinforcement learning problem and turns them into supervised learning tasks (where the right answer is known).
• This helps speed up learning and makes training more stable.

While the full details of this method aren't covered in this chapter, you can check the original paper [286] or watch the robot in action in a demo video mentioned there.

Summary of Key Points

Aspect	Details
Tasks Learned	Hanger on rack, shape sorter, hammering, bottle cap
Robot Type	Real physical robot with camera
Input	RGB camera image + robot state (joints, angles, etc.)
Network Type	7-layer CNN + fully connected layers
Unique Layer	Spatial Softmax for precise 2D position of features
Output	7 joint torques (actions)
Learning Method	Reinforcement learning + guided policy search
Total Parameters	About 92,000 (mostly in CNN layers)

9.7.3 Building Conversational Systems: Deep Learning for Chat bots

What Are Chatbots?

Chatbots (also called conversational systems or dialog systems) are computer programs designed to talk with humans using natural language. The goal of a chatbot is to have smooth, meaningful conversations with people about different topics.

However, we're still far from creating a chatbot that can talk like a real human in all situations. But there has been good progress in building chatbots for specific tasks, like:

• Helping people shop online
• Assisting with bookings
• Negotiating simple deals

A popular example is Apple’s Siri, which is a general-purpose assistant. Siri can talk about many things, but even Siri sometimes gives funny or wrong answers to common questions. This shows that general chatbots still struggle with human-like conversations.

Instead, we often focus on closed-domain chatbots. These are chatbots made for a specific job, like negotiation or customer support. Because their scope is narrow, they can be trained more effectively.

🔹 Facebook’s Negotiation Chatbot – A Case Study

Facebook built a chatbot trained to negotiate in a specific setting. This was a closed-domain chatbot, meaning it was only built for negotiation tasks.

 The Setup:

Two chatbots (or agents) are shown a set of items such as:

• 2 books
• 1 hat
• 3 balls

Each agent is assigned secret values for each type of item. For example:

• Agent A might value a hat the most.
• Agent B might value books and balls more.

The goal is to negotiate and divide the items. But:

• They don’t know each other’s item values
• The total value of items for each agent is fixed (like 10 points)
• Some items have value to both agents
• After 10 turns, they can end with no agreement, getting 0 points

This setup creates a realistic negotiation problem, where agents must learn to:

• Talk
• Bargain
• Strategically request items

🔹 How Was the Chatbot Trained?

The training combined two techniques:

1. Supervised Learning – The chatbot learns from real human conversation data.
2. Reinforcement Learning (RL) – The chatbot learns by trying, getting rewards when it negotiates successfully.

Supervised Learning:

• Facebook used Amazon Mechanical Turk to collect 5808 real dialogs.
• Each dialog had different item values and splits.
• These dialogs were used to teach the chatbot how people usually negotiate.

Reinforcement Learning:

• Once the bot had basic skills from supervised learning, it practiced by negotiating with itself.
• This is called self-play.
• Each time, it got a reward based on the final value of items it received.
• The goal was to maximize this reward over many dialogs.

But there was a problem: If both agents used RL, they started creating strange, non-human language. So, to fix this:

• One agent was kept as a supervised model (which talks like a human),
• The other used reinforcement learning (which improves negotiation strategy).

🔹 How Did the Chatbot Generate Dialog?

The model used a special deep learning structure called GRU (Gated Recurrent Unit), which is good for processing sequences like sentences.

It had:

1. A GRU to understand item values.
2. A GRU to generate sentences (dialog).
3. A forward and backward GRU to generate the final item split.

All these were connected end-to-end in a neural network.

🔹 Dialog Roll-Out (Planning):

Instead of picking a random sentence, the chatbot:

1. Created many possible sentences.
2. Simulated full conversations for each one.
3. Calculated the expected reward for each.
4. Selected the sentence with the highest reward.

This helped in choosing smarter replies.

🔹 What Did They Find?

Some key results:

• Chatbots trained only with supervised learning gave up easily and accepted unfair deals.
• Chatbots using reinforcement learning were more persistent, trying to get the best possible deal.
• These bots even used human-like tactics, such as:
• Pretending to care about an item (to trade it for something more valuable later).

Summary:

Concept	Explanation
Chatbot	A system that talks with people using natural language
Closed-domain	Focused on one task (e.g., negotiation)
Supervised Learning	Bot learns from human conversation data
Reinforcement Learning	Bot learns by trial-and-error and reward
Self-play	Bot talks with itself to learn better strategies
Dialog Roll-Out	Bot simulates different replies and picks the best one
GRU	A type of deep learning unit used for handling sequences

9.7.4 Self-Driving Cars

Self-Driving Car Case Study – Explained Simply

Self-driving cars are a great example of how artificial intelligence and learning from experience (like humans) can be applied to real-life tasks.

Just like the walking robot example, a self-driving car also needs to go from Point A to Point B safely. The main challenge? It must avoid accidents and handle different road situations like turns, signals, bumps, and other vehicles.

 The Setup: Sensors and Data

Self-driving cars are usually equipped with many sensors, like:

• Cameras (for visual input)
• Microphones (for audio input)
• Proximity sensors (to sense how close things are)
• Motion sensors (to understand movement)

All these sensors help the car "observe" the environment—just like humans use their eyes and ears.

Why Reinforcement Learning?

Driving is hard to write exact rules for, because:

• Every road is different.
• There are unexpected events (like a person crossing suddenly or a pothole).

But it’s easy to know when a car is driving well—no accidents, smooth travel, and reaching the destination.

That’s why Reinforcement Learning (RL) is suitable:
The car gets a reward if it makes progress without errors, and gets punished (or less reward) if it does something wrong like swerving or stopping unnecessarily.

A Simplified System: One Camera

Although real self-driving cars use many cameras and sensors, researchers tested a simplified version using only one front-facing camera.

Surprisingly, even this simple setup could work well when combined with machine learning!

This idea was inspired by a system from 1989 called ALVINN (Autonomous Land Vehicle in a Neural Network). The main improvements today come from:

• More data
• Better hardware (faster computers)
• Advanced algorithms like convolutional neural networks (CNNs)

Training the Neural Network: Supervised + Imitation Learning

To train the system, researchers drove the car on many different roads:

• City streets
• Highways
• Curvy paths
• Roads in different weather conditions

During training, two extra cameras were also used (placed at slight angles) to capture what happens when the car is a bit off-center or tilted. These were not used for actual driving decisions, but just to create more training data (called data augmentation).

What Did the Neural Network Learn?

The network learned to predict the steering angle—how much to turn left or right—by comparing its own output with the human driver’s command.

This is known as imitation learning:
The machine tries to “copy” what the expert human does.

This part is more like supervised learning (because it uses correct answers), but it helps kickstart reinforcement learning, which improves through experience later.

Network Architecture (Simplified)

The neural network used was quite powerful:

• 9 layers total:
• 1 normalization layer (to adjust input range)
• 5 convolutional layers (to detect features from images)
• 3 fully connected layers (to make the final decision)
• First layer used 5x5 filters with stride 2 (to reduce size quickly)
• Later used 3x3 filters for detailed feature learning
• Final output: a single value representing the steering command
• Total: 27 million connections and 250,000 parameters

Testing and Results

After training, the car was tested in two ways:

1. In a computer simulation
2. On real roads, with a human backup driver always ready

The results were impressive:

• The car drove autonomously 98% of the time
• Human intervention was needed only 2% of the time

A video of this test was also made available online.

Visualization of Learning

Researchers looked into what the CNN "saw" during driving:

• On normal roads, the network clearly learned road boundaries and important driving features.
• On forest roads, the network got confused (because it never learned to detect tree types—only road features).
• This shows that the network only learns what’s important for the specific goal (driving), unlike general-purpose models like ImageNet-trained CNNs.

Key Takeaways for Students

• Self-driving cars benefit from both supervised and reinforcement learning.
• Imitation learning helps machines copy humans initially.
• Convolutional Neural Networks (CNNs) are great for image-based decision-making.
• Data, compute power, and good training are critical for success.
• Reinforcement learning becomes useful when judging results is easy, but writing rules is hard (like in driving).

9.7.5 Inferring Neural Architectures with Reinforcement Learning

 Topic: Using Reinforcement Learning to Design Neural Network Architectures (Case Study)

What’s the Main Idea?

We are using reinforcement learning (RL) to automatically design a convolutional neural network (CNN) — for example, to classify images from a dataset like CIFAR-10 (a popular dataset of small colored images from 10 categories like dogs, cats, trucks, etc.).

Instead of a human designing the CNN manually, a special neural network called the controller learns how to build it by trial and error — just like how a robot learns to walk!

The Controller Network

Think of the controller network like an architect. Its job is to decide how to build the CNN, layer by layer.

The controller is a recurrent neural network (RNN) — that means it can remember what it decided earlier and use that memory to decide what comes next.

For each layer in the CNN (called the child network), the controller decides:

• How many filters (also called kernels) to use
• The height and width of each filter
• The stride (how much the filter moves across the image)
• And so on…

Since the design decisions are made sequentially, an RNN is a good choice — it helps remember earlier decisions while making new ones.

Building the Child Network

Once the controller gives all the layer settings, a child CNN is built using those choices.

This child CNN is then trained on the CIFAR-10 dataset to learn how well it performs (i.e., how accurately it can classify the images).

Getting the Reward

After training, the accuracy of the child network on a validation set is used as the reward for the controller.

• If the child network performs well → high reward
• If it performs poorly → low reward

This reward helps the controller learn which types of architectures are good.

How Learning Happens: The REINFORCE Algorithm

The controller network is trained using a reinforcement learning algorithm called REINFORCE.

• The controller’s goal is to maximize reward over time.
• Using the REINFORCE algorithm, the controller adjusts its internal weights to make better architectural decisions in the future.

So, the more it learns, the better it becomes at designing powerful CNNs.

Discrete Tokens

The controller doesn’t generate actual numbers like 5 or 7. It uses discrete tokens like “filter size = 3x3”, “stride = 2”, etc., chosen from a predefined set of options.

This fits nicely with reinforcement learning, which often deals with discrete action spaces (like “move left”, “move right”).

Gradual Learning: From Shallow to Deep Networks

In the beginning, the controller builds small CNNs (with few layers). As training goes on, it is allowed to build deeper networks — this helps it learn step by step without getting overwhelmed.

So over time, it goes from learning simple designs to creating deep, complex CNNs.

Extra Tricks

The method also includes some smart tricks:

• Parallel training: Many child networks are trained at the same time to speed things up.
• Skip connections: The controller can learn to add advanced structures like ResNet-style shortcuts, which help training deep networks faster.

Summary

Concept	Explanation
Controller Network	An RNN that decides how to build a CNN (the child network)
Child Network	The CNN designed by the controller, trained to classify images
Reward Signal	Accuracy of the child network on a validation dataset
Training Method	Reinforcement learning using the REINFORCE algorithm
Why RNN for Controller?	Because decisions are made one after the other (sequence)
Why Discrete Tokens?	Easier for reinforcement learning than continuous values
Gradual Increase in Layers	Starts simple, adds complexity over time
Optimization Tricks	Parallelism, skip connections, etc.

This is an excellent real-world case showing how reinforcement learning can be used not only to control actions (like robots or cars) but also to automatically design neural networks, saving time and possibly finding better models than humans can design manually.