My name is Andre Fenton.
I'm a professor of Neuroscience at New York University.
I also co-direct the Neural Systems and Behavior Course
at Marine Biological Laboratories in Woods Hole.
And I founded a company, Bio-Signal Group,
that makes medical devices that helps patients and doctors
handle neurological emergencies.
The focus of my research program is memory
and what I'm going to give you today is,
in three parts,
a series of lectures that first talk about memory in general
and how we can understand
the neurobiological underpinnings of memory,
as well as some of the research
that we've done to actually explore and elucidate
some of the fundamental mechanisms
that maintain memory in our brain.
So, let's begin.
This is a picture of some of the people in my laboratory
and one of the installations in my laboratory.
What I'd like to point out is what I'm about to tell you,
is that the work done by a large number of people,
some of them in my laboratory, some of them in other laboratories,
and this work spans several decades of human effort.
I'm going to focus on memory
and one of the things that we are interested in, in particular,
in the laboratory, is understanding the concept of knowledge or cognition.
That's the set of the mental processes
and our abilities that are related to memory,
if you want to say it generally.
And you can see here in this cartoon
that we have, for example, five senses
and we can collect information from those senses
and we can process information and generate new knowledge.
And, in particular, if you're interested, like I am,
in understanding cognitive information processing,
it's important to emphasize and to study,
to focus in one way or another,
on the kinds of information that arise
from these internal brain operations, from internal processes.
And it would be best if you could demonstrate to yourself
and be confident
that that information could never have been directly experienced or directly sensed.
And that's the kind of information that we will focus on,
as you'll see in a number of minutes.
I want to also point out that
that's not to say that if you were to study sensation,
the kinds of the things that you can collect with your senses,
that you wouldn't be able to study cognitive information processing,
but it becomes ambiguous as to whether
you're studying these internally generated notions of the world
or you're simply detecting those notions of the world.
This is one of my favorite sayings:
"What we think we become".
The Buddha is reported to have said this
and what's interesting about this idea, if you think about it,
it's a rather remarkable statement.
What we think is not what we are
or what we have been,
but what will determine our future.
And, in fact, the knowledge that I'm going to talk about today,
what we understand about the neurobiology of memory
and perception,
actually has a lot to say about the validity of this idea.
So, let's begin.
Let's talk about memory from a cognitive-psychological point of view.
Here's an experiment done by Bartlett quite some time ago
that you can actually do at home at your next dinner party, for example.
What he did was that he drew a portrait of a person,
you can see there in the upper-left,
and he drew that portrait and he said to someone,
hey, can you take a look at that, study it?
I'm going to ask you to reproduce it from memory.
And so that's what reproduction 1 is.
And then he took reproduction 1
and he asked someone else to take a look at that, remember it,
and reproduce that from memory.
That's reproduction number 2.
And so on and so on.
And what's interesting is that this abstract figure converged,
and it reliably converges,
to something recognizable.
As you can see, with reproduction number 9,
the form of a face, if you will,
a human face.
And what's remarkable about this is two things.
The first thing is it shows that
our memories aren't all that good, okay?
None of these reproductions are actually very accurate.
The second thing, I think, is the most important and salient point, here,
and that's that we're not poor in our memory in random, undirected ways, right?
The mistakes converge, if you will,
on a particular expectation or prescription, if you will,
an internal idea of what that image might have been.
And this is one of the early examples,
and pretty compelling examples,
that memory is not something that we simply
take snapshots from our experience and can reproduce.
Rather, memory is something that is active
and we reconstruct
-- we take the information in our minds and our brains
that we collect from experience,
and we reinterpret and construct that information
in order to express our memories and build our knowledge.
And so you can think about this...
this is an equation that Endel Tulving
has done interesting experiments to support...
when we talk about memory,
we really look for what information someone has recalled, right?
And that's approximately, you would imagine,
equal to what has been stored, right?
What you've actually experienced.
But Endel Tulving points out that, in fact,
it's also strongly dependent on what information
was used to actually trigger the recollection.
And, not only that, it's strongly dependent
what you recollect
on what your actual current mindset is,
what actually is of interest to you,
what you happen to be talking about.
There are many examples of this.
And so... further evidence that memory
is a reconstructive process.
And I'm very interested in understanding
the neurobiology that underlies this process.
And so the question really is then,
how does memory in the brain work?
The brain is a very complex system,
one of the most complex systems, some people will argue,
in the universe.
And in order to understand something like memory,
or any brain process, in fact,
one has to recognize that the brain operates
at many levels and those levels interact.
What do I mean by that?
Here's an animation, you can see, of a brain.
We can slice through the brain and you can see these false-colored images of the brain
that would show you different portions of the brain.
These false-colored images, now, show you
what you might recognize from the media: functional imaging,
where hotter colors represent information or activation in that part of the brain.
And now we're zooming in,
using a magnifying glass or a powerful microscope,
and if you zoom in far enough you can recognize that the brain is
made up of about 100 billion neurons,
and now you can see that those neurons are electrically active.
They're sending electrical signals
along their processes and communicating
from one neuron to the next.
This means that there are trillions of communication opportunities
within the brain,
and if we look within any one of these neurons, you can see it's more or less like a chemical factory
-- these are thousands of molecules,
different species of molecules and ions,
that move around and are regulated by proteins
and the various intracellular operations
that control the behavior, if you will,
of these individual neurons.
And to understand memory, we have to think about it at all of these levels
and try to understand the rules
of how these interactions
across all of these levels operate.
We haven't worked all of that out yet,
but I'm going to give you, across this series of lectures,
some insight into what we think
about the neurobiological basis of memory across these levels.
So, just to summarize where we are so far...
if we want to understand anything about brain function,
anything about behavior,
you have to appropriately think about this at multiple levels.
You can imagine the individual level
-- I can watch your behavior.
That's obviously going to be related to and influenced
by your social stature, your social status,
who you happen to be interacting with at one moment or another,
what you're capable of doing is certainly
influenced by your brain and what capabilities
your brain happens to have.
But what capabilities your brain happens to have
is determined by the neural networks in your brain and their operations.
Those neural networks are themselves determined
by individual cells that are embedded in those neural networks,
and the individuals cells in those neural networks
are actually influenced by their neighboring cells,
so you'll notice the two-way arrows
between all of these different levels.
The level at 6 o'clock corresponds to the synapse
-- we'll spend a lot of time focusing on that.
The synapse is just one part of the membrane of the cell
where there are proteins embedded in this lipid bilayer
that many biophysicists study.
This is actually controlled in very strong ways
by the DNA and the molecular features
that emerge from DNA being transcribed to RNA,
being [translated] to protein,
and that itself, the genetic basis of a neural system,
is obviously, through this set of interactions,
also capable of determining how a social group of animals,
people, insects, are able to interact.
So, we'll try to understand the basis of memory,
not at all of these levels,
but it's important to realize that they interact amongst each other.
We're going to focus now,
because my training and focus is in electrophysiology...
we're going to focus on neural activity.
And I'd like to show you how neural activity,
in many ways,
underlies our memories and thoughts.
And by neural activity,
what I mean is the action potential electric discharge
that neurons generate.
I'll show you a little video from a research program
out at UCLA
of electrical activity in the human brain.
And so, this is a movie that was made by
a neurosurgical team after implanting some electrodes
into the medial temporal lobe
-- that's this portion over here on the sides of the brain --
of a subject who happens to have chronic epilepsy.
Their epilepsy is so bad that it can't be
controlled appropriately by drugs
and the parts of the brain that are generating the seizures
have been difficult to identify in other means.
And so this patient is due for
neurosurgical removal of the offending, if you will,
part of the brain that's generating the seizures,
but the neurosurgical team doesn't know
which exact part of the brain is making these seizures.
So what they've done instead is implanted some electrodes
to try and, if you will,
eavesdrop on where the seizures actually get generated.
And in the meanwhile, they're able to record
not only seizures but the electrical activity
from single neurons in this person's brain.
And they've taken the opportunity,
while this patient is in the hospital for approximately a week,
to show the patient a bunch of video clips
and record the electrical activity from the brain.
And so what we're going to see here is
the firing of a single neuron in the medial temporal lobe of this patient,
while this patient is watching these different clips.
So, you can see, you'll be hearing these little clicks.
These are the activity of neurons
and you hear a big burst of firing
that corresponds to this one neuron being
very excited, very active.
It doesn't seem to care about the titanic,
the neuron seems to be uninterested in the stock exchange,
the neuron seems to be very excited by the Simpsons.
So, this neuron, in some sense,
is tuned to be representing the Simpsons,
or at least reflecting the fact that the patient
is watching the Simpsons.
It doesn't care about Martin Luther King, the pyramids,
Sex and the City doesn't do it for this neuron,
Marilyn Monroe... nothing,
the Titanic... not even that,
Martin Luther King again, second try... nada,
but to The Simpsons this neuron becomes hyper, okay?  It becomes activated.  And so that's pretty interesting,
but what's even more interesting is
if we record from this neuron
while the patient isn't watching anything
and is just remembering the video clips
that the patient was shown...
and so, here it is.
The neuron's activity while the patient
is recollecting something about New York,
something about the Hollywood Sign,
the neuron isn't active,
the neuron begins to fire,
and then the patient recognizes or remembers The Simpsons.
And so what's remarkable about this is there's very little difference
between the activity of this neuron
and other neurons in the brain
when the patient first has the direct experience of that thing
and when the patient recollects that thing.
From the point of view of the brain and from the point of view of the neurons,
there's very little distinction between the actual experience
and a recollection of that experience.
So, we can study the electrical activity in the rodent brain, okay?
With much better precision and detail,
we can do much better controlled experiments
and we'll talk about some of those experiments.
So, here's an example.
One of the cells I recorded when I was a graduate student:
a head-direction cell.
It should be obvious when we watch
a rat walking around in a cylinder,
and if you listen now,
you should be able to hear the activity of this neuron firing.
And it tends to fire when the animal's head
is pointed in that particular direction.
The neuron is quite silent
and, as the animal turns its head in that direction,
aimed at about, say, 11 o'clock,
the neuron becomes active.  And so that's quite remarkable.
Here's one neuron of millions, right,
that if we are able to eavesdrop on its activity,
we can see that it's tuned to
a very specific aspect of the animal's behavior
-- which direction the animal's head is pointing --
and we would call this cell a head-direction
or head-direction tuned cell.
There are other kinds of cells in this part of the brain,
which is the hippocampal area,
a place cell, for example.
And a place cell...
it's another rat walking around a similar environment.
This place cell, you can hear its activity
rather strongly when the animal is in this region of the space.
When it's elsewhere, the cell fires much less.
And again... now, this is one of, again,
millions of cells in the rat's brain,
but this one is tuned to
this particular part of the spatial world
that's accessible to the rat.
Now, those of you who are philosophical
might recognize that what I just showed you is rather remarkable:
single neurons in the brain seem to know about aspects of space.
And if you're familiar with your Immanuel Kant,
you'll recognize that he argued
so convincingly that it's changed
how Western thought has developed...
he argued that there is no such thing as absolute space.
We all recognize and agree, in more or less the same terms,
about where we happen to be
and where other things are in space,
but we can't know space unto itself.
And what he argues is that our experiences,
from evaluating space and also time...
that our experiences must be structured in some way, right,
by the development and the operations of our minds,
which, in today's terms,
we would say by the neural circuitry
that defines our brain and how our brain generates knowledge,
the kinds of things that we can talk about in terms of space.
And this is not trivial,
in fact it was recently recognized
with the Nobel Prize for John O'Keefe,
who first discovered place cells in the rat hippocampus,
that kind of cell I just showed you,
as well as the Mosers, from Norway,
who discovered a different kind of cell, a grid cell,
that seems to know something about
the distance an animal might have moved in an environment.
The head-direction cells were recorded first...
for the first time and discovered by one of
my mentors, Jim Ranck.
So, I've now shown you single neurons
-- a Simpsons cell, a head-direction cell, and a place cell --
that seem to code aspects of the external world.
But what kind of a code is that?
What kind of a neural code am I talking about?
Well, let's talk about codes in general.
There are at least two kinds of codes you can think about.
On the left, here, is a traffic light,
your standard traffic light.
The green light unambiguously means "GO";
it's dedicated to signaling that one message.
But you can also signal "GO" in a different way,
in a different form of traffic light,
with this pattern of lights, and the pattern of lights...
no particular light, if you make the analogy between the lights and a neuron...
no particular light is actually meaningful to the message.
It's the collective pattern
across all of the light elements that sends the message
to either stop or go.
And we call that ensemble coding.
Perhaps the most intuitive way to think about an ensemble code
is to think about your jumbotron that you might find at a sports stadium,
for example, this.
You'll notice that the message is very clear,
but there's no light bulb here that's particularly devoted
or dedicated or even meaningful to the message.
Right?
The message is encoded across the collection of neurons
in the brain and of lights in the jumbotron.
They don't even all have to be on at the same time,
and what's important is
not just which lights are on but also which lights are off.
And so what kind of a code is the hippocampus?
This is work done by these two individuals from my laboratory in place cells.
And so, we decided to ask this question:
What kind of a code is the hippocampus code for location?
And in doing so, what we did is
what would seem a very simple experiment.
We recorded the rat in a cylinder, just like I showed you earlier,
as it walked around,
and we recorded several place cells all at the same time.
We also recorded those same cells
while the rat was walking in a much larger environment,
the one that enclosed the cylinder.
And in particular, the cylinder is just short of a meter in its diameter
and the bigger chamber is about a meter and a half on the side.
And so when you record place cells, what do you see?
Well, if you record them just in the cylinder...
you let the animal walk around for 10 or 15 minutes,
like I've shown you before,
and you can see cell 1, for example,
has a place where it likes to discharge...
let's say that's close to 8 o'clock,
cell 2 like to fire near 11 o'clock,
cell 3 somewhere between 12 and 1 o'clock,
and so on.
Each cell has a different preferred part of the world
that it is encoding.
And if we take the animal away or do something else for an hour
and bring it back to the cylinder,
the bottom row here... you can see that the same pattern emerges again.
These cells you can think about as encoding, one at a time,
a particular location in space,
and if you were to consider them individuals
that would individually vote, if you will,
by the strength of their voice,
the strength of their activity,
which location the animal would be in,
it would be fairly easy to work out where the animal was
by the loudest, if you will,
contingent of cells that were active at one moment.
But that kind of coding, that dedicated coding,
doesn't work when you allow the animal
to walk in the larger environment.
Here you can see that each cell
actually fires in multiple locations
in this environment
and we can, for example, look at cell number 4.
Cell number 4, there, is firing in two locations,
and if you were to listen just to cell number 4
and ask, where is it voting?
You would have to guess between one location and another.
And any kind of averaging you did across these different cells
would take you away from
cell 4's actual place that it's representing where the animal is.
And in fact this becomes trivial
as an ensemble code
because every location in this environment
is encoded by a different pattern -- think of the jumbotron.
A different message corresponding to a location in space
is encoded by the pattern across all of these cells
at every moment in time.
And that's relatively easy to demonstrate
in this almost trivial example...
I can take the same activity and I can think about it differently.
I can think about each cell
as one of these balls in the little plot that I've made
and the red plot, let's imagine,
corresponds to the pattern of activity
during a short moment of time across all of the cells,
and it corresponds to a brain state
that would represent a location in the cylinder.
And alternatively,
when the animal is in the big chamber,
a different set of cells, but an overlapping cells,
a different patterns of light bulbs, if you will, i
n the jumbotron example,
would become active and define a different brain state.
And these two states of activity
can be easily defined
by looking at the activity of a number of cells.
Shown here, every tick
corresponds to the activity of one of these neurons,
each row corresponds to a particular cell,
and in this example I'm showing you
15 cells recorded at the same time.
Remember, there's a few 100,000
that are likely active at any moment in the actual brain.
And if you squint you can almost imagine
you can see something like a bar code
that represents the cylinder,
and a distinct bar code that represents the chamber.
And the first bar code for the cylinder
reemerges when the animal goes back into the cylinder.
So, we can actually quantify this very easily.
Instead of looking at the animal's location on average in space,
what we can do is take the activity of these cells,
the 15 cells shown here,
and we can take little slivers of time and count.
How much activity did cell 1 generate,
cell 2 generate, and so on?
And so, for each moment,
you can describe a pattern of the activity across the network of cells.
And what you can see,
what's shown here in the colorful plot,
is how related or correlated the activity is
during one moment of time, on the x axis,
and another moment of time, on the y axis.
And if you look at the diagonal,
that's a perfect correlation of 1.
And for the first 15 minutes I've collected the data, here,
while the animal is in the cylinder for the first time.
And you can see the high correlations,
the red colors,
correspond to the activity being stable of self-similar as the animal moves around in space.
But between time 15 and 30
I've used the data while the animal was in the chamber,
and it's the same cells, they're still active,
but they are uncorrelated or even negative correlated
-- anti-related --
to the activity while the animal was in the cylinder.
And, for the last 15 minutes,
I've taken the data when the animal was returned to the cylinder
and the activity is reinstated.
So, you can see that it's the same cells,
it's the same network of the brain,
but it's able to configure itself, dynamically,
into different patterns of activity that represent
different memories, different kinds of information, and such.
And so the brain, at least this part of the brain,
and I would argue most parts of the brain,
most neural circuits,
signal information using these dynamical ensemble states,
not just the hippocampus.
So, let's summarize.
What have we talked about so far?
We talked about memory as being a reconstructive process,
not a reproductive process.
When you experience the world, you
collect information and you store that.
When you recollect that information,
you actually actively build up your knowledge
that you express as memory.
It's not simply a recording that's replayed.
Memory, like all the functions in the brain,
is really operating at multiple levels
of biological organization
and they interrelate.
That's important because you can't understand
and you can't treat memory
and you can't improve memory
by understanding or dealing with it at one level or another.
You have to take into account the genetic, the molecular,
the cellular, the neural circuit,
and the behavioral... in fact,
the social consequences for such a system.
We've talked about the code for spatial information in the brain,
but that's really just an example for most information,
most cognitive information in fact,
and that information is encoded as an ensemble,
not in these dedicated ways.
And what that means is,
if you want to do the trick of mind reading,
if you'd like to decode  information from an ensemble of neurons
or any ensemble code, in fact
-- the jumbotron as the example --
you can't analyze it one lightbulb at a time.
You actually have to record
a large number of those lightbulbs
and it requires the analytical techniques
to understand the timing and the organization in time
of the activity across multiple elements,
millions in the case of the brain,
hundreds of thousands if you downsamples,
and certainly hundreds.
And this information can't be extracted
from an ensemble code by just average across individuals;
it requires dedicated effort
to collect the information across many examples
or elements of such a system,
and in fact this is the goal
of what's known by the acronym BRAIN
-- Brain Research through Advancing Innovative Neurotechnologies --
which is an initiative that's been spearheaded
and sanctioned by the White House,
and I've provided some links so that you can learn about that,
where there's an effort amongst neuroscientists to
develop new tools so that we can drive
and inspire our increased understanding of how
to understand and record and interpret these ensemble codes
that operate at multiple levels of biology.
So, I'll close now
by thanking and acknowledging the many people in my laboratory
currently and in the past,
and my many collaborators who've contributed to the research
that we're doing.
And very importantly,
the various sources of funding that have contributed to this work over the decades.
