fast.ai reference
fast.aicoursematerialbook
Search…
Fast.ai reference
How to run the code
PyTorch
Jupyter Notebook
Computer Vision
Tabular data
NLP
Reinforcement Learning
Overfitting and Underfitting
Contribute
FEEDBACK & CORRECTION
Powered By GitBook
NLP
Natural Language Processing
Natural Language Processing (NLP) is the field in which information is extracted from spoken or written language.
NLP is special because the meaning of a word is heavily context dependent:
  • The bird is a crane.
  • A crane is used in industry to lift heavy objects.
  • I have to crane my neck to look at him.

Preserving the context

In images, shapes too are context dependent. In Computer Vision however the context is preserved with convolutions and pooling. With text, we cannot do this.
An alternative is processing the context words in addition to each target word:
This naive approach works, but is computationally intensive. Additionally, for questions of content, a few words of context is not sufficient. In the above example if asked the question "What is fun?", the model would have no way of knowing that the answer is "Deep Learning", because "fun" does not include that context.

Recurrent Neural Networks

​Recurrent Neural Networks (RNN's) attempt to solve the problem of storing context by having their own output fed back into them. By doing this for each processed word, information is preserved and passed along.
This can take various forms, LSTM's being the most popular.
A very simple NN that feeds the output of its middle layer to a "recurrent" layer, and then gets it back
​
Previous
Fully connected layer
Next
Reinforcement Learning
Last modified 2yr ago
Export as PDF
Copy link
Outline
Preserving the context
Recurrent Neural Networks