gpt3 is an advanced language model which
is able to perform some language
processing tasks such as text generation
basic question answering summarizations
and other
without direct supervision literally you
tell the computer what to do and provide
some examples of desired output and the
model returns the result immediately no
coding no
fine-tuning required the idea behind
GPT 3 is very simple to train the model
on a massive amount of text and then see
what will it learn the architecture used
in gpt3 is called the transformer and it was
well known in the machine learning community
In simple words, it works in a way that
the word in a sentence is masked and
then predicted using weights assigned to
surrounding words
such iteration is repeated for each word
so nothing new was created the
innovation of gpt3 3 lies within the
data which was used for training
to be more precise the volume of data on
which it was trained
gpt3 was trained on 750
gigabytes of data and as a result it has
much more parameters
175 billion parameters in comparison
with
1.5 billion parameters are in the old
version to
understand how large the data set was
imagin that one text file consists of
around
seven hundred thousand pages per one
gigabyte the data corpus contains
petabytes of data collected over eight
years of web crawling 8 million
web pages books and english wikipedia
 
 
processing such share amount of data it
would take
114 years of training on a single server
using
for example amazon web services and in
that case it would cost
over 1 million us dollars but if you
have access to a super computer
it would take you 24 days of training
but it would cost much more
 
gpt3 was very successful in solving the
the problem of predicting a word completing
the sentence
as you can see from this plot gpt3
outperformed state of the art
model on lambada dataset
but it's still quite far from the level
of the human ability to predict
the next word what is more interesting
that the accuracy which with which
gpt 3 was able to predict a word
was enough to solve so many different
problems
such as generating news generating
poems generating novels generating
completing excel spreadsheets and other
official documents
gpt 3 was trained on English datasets
but seven percent of data was in other
languages translation
tasks however it did not outperform
state-of-the-art model what is more
interesting
is that translation
may be used for different types of data
it may also be effective
in translating text
to the code or translating
the particular style of the language to
another style for example
legal language to natural language
gpt3 was quite successful
in doing it according to
some publicly available demos
gpt3 outperformed a fine-tuned
state
of the art model for question answering
task
and as you can see from the previous graph
it uh scaled with the number of
parameters as such we can predict that
the more parameters will be used in the
model
the better question answering task
will be solved
gpt3 outperformed state-of-the-art
solutions for common sense reasoning
but it wasn't still good for
natural language inference tasks where
the reasoning is more complex and involves
different levels
of logical inference
gpt3 is not ideal it's still quite far
from the human ability to process the
data
moreover it didn't outperform
state-of-the-art models on some tasks
the weakness of gpt-3 may be caused by
both
simple architecture and structural
constraints, not all information
is expressed in written form human
perception of the world is not limited
to the content of the book of our pages
as such the gap between the way how
human and computer understand and
process information
cannot be eliminated
gpt 3 was created by a company called
openai
this company expressed some concerns and
possibility of harmful application of
jpg3 to name a few discrimination
fraud misinformation and prejudice
as such the results of the training were
not publicly released and the access was
open upon request to selected users only
well it's open to discussion who shall
be in control of such powerful tool
there are no doubts that the progress
cannot be stopped
thank you for watching this video until
the end don't forget to subscribe and like
this video
