BAI701-Deep Learning- Module 4 Notes

 

Module 4

Recurrent and Recursive Neural Networks Unfolding Computational Graphs, Recurrent Neural Network, Bidirectional RNNs, Deep Recurrent Networks, Recursive Neural Networks, The Long Short-Term Memory.Gated RNNs.

Text Book – 2: 10.1-10.3, 10.5, 10.6, 10.10

10. Recurrent neural networks

What is a Recurrent Neural Network (RNN)?

A Recurrent Neural Network (RNN) is a special type of neural network designed to work with sequential data—data where the order matters.

Examples of sequential data include:

• Sentences (where word order matters)
• Time series data (like stock prices)
• Audio signals (where each moment depends on the previous ones)

Just like Convolutional Neural Networks (CNNs) are made for images (grids of pixels), RNNs are made for sequences of values.

Why Are RNNs Special?

In a regular neural network (like a feedforward neural network):

• Each input is processed independently.
• It’s difficult to use it for data where previous inputs affect the current prediction.

But in RNNs:

• Previous information is passed forward through something called a "hidden state".
• This means the network can "remember" what happened before.

So, RNNs are good at handling things where context is important:

For example, in the sentence:

• “I went to Nepal in 2009”
• “In 2009, I went to Nepal.”
  Even though "2009" appears at different positions, an RNN can still learn that it refers to the year of the event.

How Does RNN Work?

At each time step t, the RNN:

1. Takes an input vector x(t) (like a word in a sentence).
2. Combines it with the previous hidden state (memory of the past).
3. Produces:
• An output (like a prediction)
• A new hidden state, passed on to the next time step.

This repeating structure is like a loop:

x(1) → h(1)

        ↓

x(2) → h(2)

        ↓

x(3) → h(3)

        ↓

...   ...

The same function (with the same weights) is used at each time step. This is called parameter sharing.

Parameter Sharing – Why It Matters?

Imagine you have 3 sentences of different lengths:

• "He runs"
• "She is running"
• "They have been running fast"

If your model had different parameters for each word position, it would:

• Need to learn language rules separately for each position.
• Struggle with new sentence lengths.

Instead, RNNs share parameters at every step, so:

• The same rules apply to each time step.
• The model can generalize to longer or shorter sequences.

This is like teaching one rule for "verb tense" and applying it wherever needed, no matter where the verb is in the sentence.

RNN vs 1D Convolution

• A 1D Convolutional Neural Network also uses parameter sharing, but it looks at only a small set of nearby inputs (a local window).
• An RNN, in contrast, builds on the entire sequence history—it remembers everything up to the current time step.

Does RNN Always Mean Real-Time?

Not necessarily. The time steps t = 1 to τ just refer to sequence positions, not real-world time. So RNNs can be used even when:

• The sequence is already complete (like in a paragraph).
• The sequence is spatial (like processing a row in an image).

Some advanced RNNs even process sequences backwards and forwards (called Bidirectional RNNs).

Cycles in Computation

In most neural networks, data flows straight from input to output (a directed acyclic graph, or DAG).

But in RNNs, there are cycles because:

• The output at one step affects the input at the next.
• This creates a loop in the computational graph.

This loop allows RNNs to carry information across time—a form of memory.

✅ Summary for Students

Feature	Feedforward Neural Net	RNN
Input type	Fixed, non-sequential	Sequence (like text or time series)
Memory of previous inputs	❌ No	✅ Yes
Handles variable-length data	❌ No	✅ Yes
Parameter Sharing	❌ No	✅ Yes (across time steps)
Example use cases	Image classification	Language modeling, time series, speech

10.1 Unfolding Computational Graphs

Understanding Recurrent Neural Networks (RNNs) and Computational Graphs

What is a Computational Graph?

A computational graph is a way to clearly show the flow of operations in a neural network. It represents how inputs, weights (parameters), and functions are used step-by-step to calculate outputs and losses (errors). It helps us understand and organize the calculations.

What is Recurrence and Unfolding?

Some systems (like time-based systems or sequences) have repetitive or looped structures. These are called recurrent systems. In neural networks, this concept is used in Recurrent Neural Networks (RNNs).

For example, let’s say we have a formula:

s(t) = f(s(t−1); θ)

Here:

• s(t) is the state of the system at time t
• s(t−1) is the state at the previous time step
• θ is a set of shared parameters (weights)
• f is a function that determines how the state changes

This is recursive because the current state depends on the previous one.

To train or analyze this structure, we unfold it.

What Does “Unfolding” Mean?

Unfolding means writing out the loop or repetition step-by-step for a certain number of time steps.

Example: If we do it for 3 steps:

s(3) = f(s(2); θ)

     = f(f(s(1); θ); θ)

Now the repetition is written as a regular sequence of functions. This forms a computational graph without loops, also called a directed acyclic graph (DAG). Every step has its own node in the graph.

Why Is This Useful?

Unfolding the recurrent function gives two big advantages:

1. Same input size at every time step:
   We don’t need a separate model for sequences of different lengths. Each step just takes the current input and the previous state.
2. Parameter sharing:
   The same function f with the same weights θ is used at every time step. This helps the model:
• Learn more efficiently
• Generalize to longer or shorter sequences than those seen during training

Unfolding with External Inputs

Sometimes the system also takes an input at each time step, like:

s(t) = f(s(t−1), x(t); θ)

Now, each step depends on:

• The previous state (s(t−1))
• The current input (x(t))

In RNNs, we often call the state h(t) instead of s(t). So the formula becomes:

h(t) = f(h(t−1), x(t); θ)

This means:

• h(t) is the hidden state (a summary of previous inputs)
• x(t) is the current input
• θ are the shared weights
• f is the function (often a neural network layer)

What Happens in an RNN?

• At each time step, the RNN updates its hidden state h(t)
• It uses h(t) to make predictions or pass information to the next step
• The same function f is applied again and again, with the same parameters

This forms a chain-like structure. If we draw it step-by-step, it looks like a long sequence of repeated blocks.

Two Ways to Draw an RNN

1. Compact (Recurrent) Diagram:
• Shows just one unit with a loop (representing repetition)
• Looks like a simple circuit
2. Unfolded Diagram:
• Shows each time step as a separate block
• Makes it easier to understand how information flows over time

 

Information Flow in RNNs

When training:

• Forward pass: Calculates outputs and loss by moving from past to future
• Backward pass: Calculates gradients (errors) from future to past using Backpropagation Through Time (BPTT)

Why Use RNNs?

RNNs are useful when:

• You want to remember previous inputs
• The input sequence length is variable
• Examples: speech recognition, text prediction, time-series forecasting

Key Takeaways

• RNNs work on sequences by remembering past data using a hidden state.
• They use the same function repeatedly over time steps, sharing parameters.
• Unfolding the recurrent computation helps visualize and train the model using traditional backpropagation.
• Compact (looped) and unfolded (step-by-step) graphs are two ways to represent the same model.

10.3 Bidirectional RNNs

Understanding Bidirectional Recurrent Neural Networks (Bidirectional RNNs)

Basic RNN Limitation: Only Looks Backward

Up to this point, the recurrent neural networks (RNNs) we've discussed only look at the past and present information to make a decision.
For example, at time step t, the RNN can access:

• Past inputs: x(1), x(2), ..., x(t-1)
• Current input: x(t)

But it cannot look into the future, such as x(t+1), x(t+2), ....

This is called a “causal” structure, because the model only considers causes from the past and not future context.

Why Is Future Information Important?

In many real-life problems, the current output depends not only on the past and present but also on the future inputs. Here are some examples:

Speech Recognition:

To identify the correct sound (phoneme) you're hearing now, it might help to know:

• What sounds come next
• Even what words follow next

Why? Because sounds can blend together (co-articulation), and meaning can depend on nearby words. Example:

• The word “write” and “right” sound the same. Only the next few words may clarify the meaning.

Handwriting Recognition:

The interpretation of a letter may depend on what comes next — some letters look similar and are disambiguated by their neighbors.

So, using only past data is not always enough!

Bidirectional RNN to the Rescue!

To solve this, researchers created the Bidirectional RNN.

What is it?

A Bidirectional RNN has two RNNs:

• One processes the sequence forward (from the start to the end)
• The other processes the sequence backward (from the end to the start)

These two RNNs work in parallel.

Each time step t now has:

• h(t): State from the forward RNN
• g(t): State from the backward RNN

The output o(t) at each time step depends on both directions:

o(t) = function of [ h(t), g(t) ]

This means the model has complete context — past, present, and future.

Benefits of Bidirectional RNNs

• Output is more accurate because it uses more context.
• No need to define a fixed “look-ahead” window.
• Especially helpful when decisions at the current point depend on what’s coming next.

Applications of Bidirectional RNNs

Bidirectional RNNs have been successfully used in:

• Handwriting recognition
• Speech recognition
• Bioinformatics (like DNA sequence analysis)

They help because these tasks often require understanding of the whole sequence, not just part of it.

Extending the Idea to Images (2D Input)

What if the input is an image instead of a sequence?

You can extend bidirectional RNNs to two dimensions (2D), where data comes not just left to right, but also top to bottom.

So, you can have four RNNs moving in four directions:

• Left → Right
• Right → Left
• Top → Bottom
• Bottom → Top

Each point (i, j) in the image grid can combine information from all directions to compute its output O(i, j).

This allows:

• Local detail understanding (like edges, textures)
• Long-distance interactions (far-away pixels can influence the output)

RNN vs CNN in Images

• CNNs (Convolutional Neural Networks) are more efficient and faster.
• But RNNs for images can:
• Capture long-distance relationships better
• Allow more flexible context usage
• This comes at a cost: RNNs are computationally heavier and slower than CNNs.

 Summary

Concept	Explanation
Causal RNNs	Use only past and present data
Bidirectional RNNs	Use both past and future data
Why needed?	Some tasks need full context (e.g., speech, handwriting)
How?	Two RNNs: one forward, one backward
Extension to 2D	Use 4 RNNs: up, down, left, right for image data
Pros	Better context and accuracy
Cons	More computation and memory usage

10.5 Deep Recurrent Networks

Computation in Recurrent Neural Networks (RNNs): Deepening the Architecture

In most Recurrent Neural Networks (RNNs), the processing can be split into three main parts or transformations:

1. Input to Hidden State: This step takes the input at the current time step and transforms it into a hidden representation.
2. Hidden to Hidden State: This part takes the hidden state from the previous time step and uses it to update the hidden state for the current time.
3. Hidden State to Output: Finally, this step transforms the current hidden state into an output.

Each of these transformations is normally done using a simple operation: a weight matrix (which is learned during training), followed by a non-linear activation function like tanh or ReLU. These are called shallow transformations, similar to one layer in a deep neural network (MLP).

Can We Make These Transformations Deeper?

Researchers asked: Can we improve performance by making each of these three parts deeper?

• Instead of just one simple layer between input and hidden, or hidden-to-hidden, or hidden-to-output, we can use a deep network (like a multi-layer perceptron - MLP).
• This was found to work well in practice by researchers like Graves (2013) and Pascanu (2014a).
• They found that deeper RNNs can learn better because they can capture more complex patterns in data.

How Can We Add Depth?

Figure 10.13 (explained in simple terms):

1. Figure 10.13a – Deep Hidden States:
   Instead of one hidden layer, we stack multiple hidden layers (like a hierarchy). The lower layers focus on raw input, and higher layers learn more abstract features.
2. Figure 10.13b – Deep Transforms in All 3 Parts:
   We can use deep networks (like MLPs) in each of the three parts: input-to-hidden, hidden-to-hidden, and hidden-to-output. This adds more depth and learning capacity.
3. Figure 10.13c – Skip Connections to Help Training:
   One problem with adding depth is that it makes training harder because the network becomes deeper in time — information has to travel through more layers, which can slow learning.
   To solve this, we can use skip connections (shortcuts that skip layers), which make it easier for gradients to flow during backpropagation. These help prevent learning from getting stuck.

Key Takeaways

• Standard RNNs are shallow in their operations.
• Making them deep by adding layers inside the input, recurrent, and output parts can help the model learn better.
• But adding depth increases the path length between time steps, which may slow learning.
• This issue can be fixed using skip connections, which shorten the learning path.

This idea of deep RNNs is similar to why deep feedforward networks (like ResNet) became popular — deeper models learn better, but we have to help them train efficiently.

10.6 Recursive Neural Networks

What Are Recursive Neural Networks?

Recursive Neural Networks (RecNNs) are a special type of neural network. They are different from Recurrent Neural Networks (RNNs), even though their names sound similar.

RNNs have a chain-like structure — they process one input after another in a straight line over time.

Recursive Neural Networks, on the other hand, have a tree-like structure. Instead of processing a sequence in a line, they combine parts of the input in a hierarchical way, like building a pyramid from the bottom up.

Where Are Recursive Neural Networks Used?

Recursive networks are useful when the input data is naturally tree-structured, such as:

• Natural Language Processing (NLP): Words in a sentence can be arranged into a parse tree (like a grammar structure).
• Computer Vision: Objects in an image can be combined based on their parts (e.g., eyes, nose → face).
• Any structured data: XML, code trees, molecule structures, etc.

Some researchers who helped develop and apply this idea include:

• Pollack (1990) – introduced the concept,
• Socher et al. (2011–2013) – applied it to NLP and vision,
• Frasconi et al. (1998) – used it for structured data,
• Bottou (2011) – discussed how recursive networks could learn structure from data.

 Why Use Recursive Networks?

Recursive networks offer one major advantage over RNNs:

• For a sequence of length τ, an RNN has a depth (number of processing steps) of τ.
• But a Recursive Network can reduce this depth to around log(τ) (logarithmic), which means it processes faster and more efficiently.

This is especially useful for long sequences, where RNNs may struggle to remember information from earlier steps.

 How Do You Build the Tree?

This is still a challenging question in practice. There are a few options:

1. Use a fixed structure like a balanced binary tree.
2. Use a structure from outside tools, such as a parser that tells how a sentence should be broken down into phrases and words.
3. Learn the structure automatically: Ideally, the model should figure out the best tree structure by itself while learning. This is an ongoing research area.

Variations in Recursive Networks

Recursive networks can come in different forms:

• Some models associate each node of the tree with both an input and a target output.
• The computation at each node doesn't have to be a simple linear operation. It can be more advanced, like:
• Tensor operations
• Bilinear forms (used to model relationships between different concepts)

These methods help capture complex relationships between elements (like how two words in a sentence interact).

Also, each input (like a word or object) is often represented using embeddings — continuous-valued vectors that capture meaning or features.

Summary

Feature	Recursive Neural Network (RecNN)
Structure	Tree (not a chain like RNN)
Useful for	NLP (parse trees), vision, structured data
Advantage	Shorter depth for long sequences → better memory
Tree building	Fixed, parser-based, or learned during training
Computation at each node	Can use advanced operations (tensor, bilinear, etc.)
Representation	Works with vector embeddings
10.10 The Long Short-Term Memory and Other Gated RNNs

What Are Gated RNNs?

Gated RNNs are a special type of Recurrent Neural Networks (RNNs) used to process sequences — like text, time series, speech, etc.

These are smarter versions of regular RNNs, built to solve two big problems that traditional RNNs often face:

1. Vanishing gradients (the network forgets past information quickly),
2. Exploding gradients (the network becomes unstable).

Gated RNNs solve these problems by controlling what information is remembered, updated, or forgotten at each step using special gates.

What Does "Gated" Mean?

Imagine a gate in real life — it either lets something pass or blocks it.

Similarly, gated RNNs use "gates" (mathematical functions) that control the flow of information in the network. These gates help the network:

• Store important information for a long time,
• Forget irrelevant information when it’s no longer needed,
• Decide what to output at each time step.

Two Famous Types of Gated RNNs:

1. LSTM (Long Short-Term Memory)
2. GRU (Gated Recurrent Unit)

How Does an LSTM Cell Work?

An LSTM cell is like a small computer inside the neural network. It has:

1. Input gate

Controls how much of the new information from the current input should be added to the memory.

2. Forget gate

Decides how much of the old memory (from the previous step) should be erased or kept.

3. Output gate

Decides what information from the memory should be output at this step.

4. Cell state

This is like a memory lane that carries information from one time step to the next, with very little change unless a gate says otherwise.

Diagram Overview (from Figure 10.16)

Here's what happens inside an LSTM cell at each time step:

• The input goes in and is processed.
• The input gate decides if this info is worth saving.
• The forget gate checks whether to remove any old memory.
• The cell state gets updated (keeps important info across time steps).
• The output gate controls what gets passed to the next layer or next time step.

All gates use a sigmoid activation (values between 0 and 1, like "yes/no" decisions).
The input signal can use other activation functions (like tanh or ReLU).

Why Is This Useful?

In many real-world tasks, we need to:

• Remember information for a long time (e.g., remembering a subject from the start of a sentence),
• Forget old details (e.g., when switching topics in a sentence),
• Update knowledge (e.g., when learning new facts).

Leaky units (used in older models) kept adding to the memory but had no good way to clear it when needed. Gated RNNs (like LSTM) fix that by learning when to forget.

Real-Life Example: Understanding a Paragraph

Let’s say the network reads this paragraph:

“John went to the shop. He bought a pen. Then he went home.”

To answer the question “Who bought a pen?”, the model must remember “John” from the first sentence and connect it with “he” in the second sentence.

An LSTM network can keep “John” in memory, thanks to its cell state and gates, while reading the rest of the sentences.

Summary Table

Feature	Gated RNNs (LSTM/GRU)
Solves	Vanishing/exploding gradient problems
Memory	Uses a long-term memory (cell state)
Gates used	Input gate, Forget gate, Output gate
Learns to	Remember, update, or forget information
Works well for	Long sentences, sequences, time series, etc.
Advantage	Better control of what is stored and erased

10.10.1 LSTM

Understanding the Core Idea Behind LSTM

In normal Recurrent Neural Networks (RNNs), when we try to pass information across many time steps, the gradients (which are needed for learning) either become too small (vanish) or too large (explode). This makes it very hard for the network to learn long-term patterns.

To solve this, researchers introduced a clever idea in LSTM networks — they added self-loops (paths that allow information to flow from one time step to the next without changing too much). This helps the gradient pass smoothly through many time steps during training, without vanishing or exploding.

Self-Loops with Gating – Smarter Memory Control

Instead of using a fixed weight (strength) for this self-loop, LSTM networks use a gate to control the weight dynamically, based on the current context or input.

• This means: Even if the LSTM has fixed parameters, it can adapt how long it remembers something, depending on the input sequence.

What is an LSTM Cell?

An LSTM cell is a special type of unit in a neural network. It replaces the regular hidden unit of a simple RNN. Unlike a basic neuron, an LSTM cell has:

1. An internal memory (state) that can keep information over time.
2. Three gates to control the flow of information:
• Forget Gate: Decides what to forget from the past.
• Input Gate: Decides what new information to add.
• Output Gate: Decides what to show as output from the memory.

Forward Pass in LSTM: Step-by-Step

Let’s see what happens at every time step t:

Why LSTM is Powerful?

• LSTM can learn when to remember and when to forget.
• It can adjust the duration for which it keeps information — useful for tasks like:
• Speech recognition
• Handwriting generation
• Machine translation
• Image captioning
• Parsing text

Optional Feature: Using Memory as Input to Gates

Sometimes, we use the memory state s^t_i as an extra input to the gates. This helps make even more precise decisions. However, it adds three more weights (one for each gate).

Performance

LSTM has shown excellent performance in:

• Synthetic datasets where long-term memory is tested.
• Real-world applications where previous RNNs failed.

Many researchers have built variations and improvements over the original LSTM to make it even better.

10.10.2 Other Gated RNNs

What Are the Important Parts of the LSTM Architecture?

Researchers asked a very practical question:
"Which parts of the LSTM are really important? Can we design other simpler architectures that still allow the network to learn what to remember and what to forget?"

To answer this, scientists developed another type of neural network called Gated Recurrent Units (GRUs).

GRU – A Simpler Version of LSTM

GRUs were introduced in 2014 by Cho and others. GRUs are similar to LSTM, but simpler.
The key idea is that instead of using separate gates like LSTM (input gate, forget gate, output gate), GRU uses only two gates:

• Update Gate (u) – Controls how much of the previous memory should be kept.
• Reset Gate (r) – Controls how much of the past should be forgotten when calculating the current state.

These gates help the network decide when to keep or forget information, based on the input sequence.

GRU Update Equation

The GRU updates the hidden state using this formula:

h_t = u_t * h_(t-1) + (1 - u_t) * new_info

Here:

• h_t is the current hidden state.
• u_t is the update gate value.
• h_(t-1) is the previous hidden state.
• new_info is the candidate state computed using the input and the reset gate.

This formula mixes the old memory and the new memory, controlled by the update gate.

How Do the Gates Work?

• The update gate (u) decides how much past information should be carried forward.
  If u_t = 1, it keeps the old memory. If u_t = 0, it forgets the old and uses the new info.
• The reset gate (r) decides how much of the past should be considered when computing the new information.
  It acts like a filter to ignore irrelevant old data.

Variations and Experiments

Researchers experimented with many variations of LSTM and GRU, like:

• Sharing reset/forget gates across multiple units.
• Using global gates for entire layers and combining with individual gates per neuron.

However, no new version was found to consistently perform better than both LSTM and GRU across many tasks.

Key Findings

1. The forget gate in LSTM is very important for learning long-term dependencies.
2. A small tweak – adding a bias of 1 to the forget gate – was found to significantly improve LSTM performance.
   This idea was suggested by Gers et al. in 2000.

Summary

• GRU is a simpler alternative to LSTM, using only two gates (update and reset).
• These gates allow the model to dynamically learn what to remember or forget.
• Despite many experiments, LSTM and GRU remain the most successful for sequence learning.
• The forget gate is one of the most crucial components in these models.