I'm Alan perelson my day jobs at Los
Alamos in the theoretical biology group
but I've been associated with the Santa
Fe Institute since 1978 both as an
external faculty member and more
recently science board member and more
to the topic I've been a collaborator
and friend of stephanie far as to
speaking tonight for the last 25 years
since some of you may not have been here
yesterday Steph gave a fantastic lecture
and was introduced and many accolades
were presented about her but I thought I
think because of limitations of time
people really didn't fully explore her
background and her expertise so just to
quickly review for those of you who
aren't here Steph got her bachelor's
degree in Santa Fe at st. John's then
went to the University of Michigan where
she got both the Masters and PhD in
computer science her thesis was done
with John Holland who's one of the
founders of the Santa Fe Institute and
in fact gave the first ghulam lecture 20
years ago after leaving the University
of Michigan she spent a few years in
industry working in the computer
industry and then decided to come back
into academic life and came to Los
Alamos and was a postdoc at the Center
for nonlinear studies which is an
institution somewhat similar to SF I
who's at Los Alamos where she came as a
postdoc that I first met her we
carpooled from Santa Fe and Steph got
interested in some of the work that I
was doing in building computer models
and mathematical models of the immune
system and I think that inspired some of
the work that you will tell you about
today when she left Los Alamos in 1990
she joined the faculty at the University
of New Mexico where she advanced through
the ranks she's moved from assistant to
associate the full professor she was
chair of the computer science department
at UNM for five years then she became a
Regents professor and more recently a
distinguished professor which i think is
their highest rank at the University of
New Mexico
for the work in computer science that
she's done she's won numerous awards
that weren't mentioned last night last
night she spoke about developing
computer systems that can automatically
fix computer bugs and for that works you
want a number of awards for best paper
or set of papers one of them was the
Manfred tall Award for Excellence in
software which you write a one in 2009
with her colleagues for this work and it
came with an award that must be special
for computer science of ten thousand and
twenty four euros which is what
basically One Killed not kill Oh bite
but kill Oh buck for those of you who
are familiar with that later that year
she won another award from the Genetic
Programming and evolvable machines group
called the u.s. gold medal award this
one came with a more substantial award
of five thousand dollars again for sort
of the best software for evolving
software and then in 2011 she won one of
the top awards in computer science
something called the Allen new Alan
Newell award and let me just read what
this awards for its presented to an
individual selected for career
contributions that have breath with in
computer science or that breach computer
science know the disciplines this award
is endowed by a prize of ten thousand
dollars and this year at one just by
herself without her her collaborators in
the award for Stepford herself you can
all read the citation on the web but it
was for fundamental paradigm changing
contributions to computer science and
biological sciences most notably
bringing together models of immune
systems automated diversity and network
of Epidemiology with significant impact
on real computer and biological research
and practice and I think during the ulam
lecturers you will really bring to for
this marriage of biology and computer
science she also mentioned last night a
lot of the economic impact of computer
bugs and types of work done in computer
science and she was invited to the Davos
economic summit last year in Switzerland
where she addressed world lead
isn't spoke about software and its
biological in biological design for for
software so without further ado I'm
hoping Steph will tell us a little bit
more about her work in biology and
computer computer science and there's
one thing I forgot in addition to
working at the level of computer science
Steph both directly and indirectly has
made some real contributions to
biological research and an influenza in
particular one of her PhD students that
I co supervised with her Derek Smith at
the time developing computer algorithms
which were you using and developing
computer models for influenza infection
those types of algorithms have now been
developed and derek is a member of the
world health organization committee that
chooses the flu vaccine every year and
some of the work that Steph supervised /
and I supervised during his PhD thesis
has evolved into using computer
algorithms to help but choose a strain
of flu that goes into the vaccine every
year so a very practical application of
a computer science staff wherever you
are I want you come on up and prevent me
from saying more great things about you
well Ellen thank you for that very
generous introduction I'm going to
repeat a little bit Alan introduced me
to immunology primarily through a series
of conversations in a carpool to Los
Alamos as he mentioned when I was a
postdoc and this carpool was really a
central part in my intellectual life at
Los Alamos and he ultimately convinced
me that immunology was more interesting
than whatever it was I thought I was
going to work on when I was there and so
then the next step was I had to learn
some immunology and so Alan gave me I
have to say endless tutorials it just
took a long time for all the big words
and all the different cell types to
start sinking in and then also one of
the algorithms that I'm going to talk
about tonight he actually proposed he
really started me on this path of
computer immune systems for security by
proposing that negative selection could
actually be turned into a computer
algorithm and so I'll talk a little bit
about that tonight and so as a world
result of this we have collaborated for
many years we even have a patent
together and Ellen thank you for these
many years of friendship and and
collaboration so the goal of complexity
science in my view I don't know if
everyone agrees with me is to discover
the common principles underlying systems
such as economies immune systems
computer networks social networks and
when we look at these networks one
common property that we see is the
emergence of malicious behavior and this
behavior arises spontaneously in
networks that that have self optimizing
processes where the individual agents
can optimize their own behavior and and
have a way well there's they're
interested in doing as well as they can
they're self-interested and they can
somehow adjust their behavior over time
and so I have a few examples up here an
example from biology is the this new
disease that's being called the middle
middle east respiratory syndrome that
was is in the Middle East and Saudi
Arabia primarily and this is the
Egyptian tomb bat that they have traced
the disease back to as being the the
most likely vector and so we have these
emerging diseases in populations there's
others that you've read about SARS for
example which is related to this one we
also in social situations we have this
kind of persistent problem of bullying
and that's a form of malicious behavior
that just kind of arises whenever we
have selfish kids or selfish adults next
to each other and then finally we have
malicious hackers who have come sort of
come along as the internet has evolved
and matured so have these malicious
hackers and the two things that that I
think really provide the the environment
for these kinds of malicious behaviors
to arise are as I mentioned before nodes
in the act in in these networks
self-interested actors self-interested
nodes in some way
and the ability to adapt their behavior
to improve their their payoff or or
fitness okay so we're going to focus on
the last of those the malicious hackers
tonight and anyone who reads the
newspaper is aware that computer
security is not very secure and might be
getting worse every day in spite of all
this great work I'm going to tell you
about tonight and as our banking and
socializing and personal lives have
moved online this is really much more
than an academic problem we're all
living in this kind of continual cycle
of innovative attack methods and I know
that you can't can't read everything on
this little graphic on the side but the
colors indicate different generations in
the evolution of computer attacks and so
those of you who are old enough to
remember floppy disks will remember the
floppy disk era when we had file
infector and boot sector viruses that
were passed around manually from
computer to computer via these floppy
disks and then when we had the the
internet came of age and people got on
the internet and started sending email
and started using the web we had the era
of worms as they're called and you might
remember Code Red and the Melissa
viruses and some of these famous email
viruses so that was the second era then
the third era really came about once
once money came to the internet when we
hit when when the era of e-commerce
really matured the we started having
these malicious websites and so in this
era I'll talk about this a little bit
more later but if you click the way
these attacks work if you go click on
the wrong website the act of clicking on
the website can cause malicious software
to be downloaded onto your computer and
finally now that we have facebook and
twitter and other social networking
for structures we have so we have a tax
that that work at that level so one
thing that's happened during this era is
that the the terminologies changed we
started out talking about viruses and
then we got worms and then a few more
things happen and it wasn't really clear
what the difference was between a virus
and a worm and now people just sort of
threw up their hands and it's all called
malware so malware is this encompassing
encompassing term and the thing about
these attacks is that every time every
time a new generation of attacks comes
along the computer security people
scurry around and we had antivirus
companies that started and then we had
new companies that started in response
to the worms and so it's this kind of
perpetual arms race between the
attackers and the defenders add many
people many people in Washington many
funders have this added are frustrated
with this and they they really just
think that we should be able to if they
put enough money into it and gave it to
the right people we could solve this
problem once and for all and my position
is that that's unlikely to happen I
think we need to learn to live with it
and manage it much in the way that
biological systems do finally another
point about the cyber security problem
is that there's a phenomenon called
Moore's law which most of you have heard
of and this is this sort of relentless
progression of computing technology
every 18 months we have double the
computing power and roughly double the
amount of memory and networking speed
networking speeds increase similarly and
so for many problems in computer science
graphics is a good example other image
processing algorithms even machine
learning those are all problems that
were not tractable 10 or 15 years ago
that are tractable now in large part
because of the increased amount of
computing power that we have but in this
particular problem that I'll be talking
about tonight that's not true every time
there's another revolution of Moore's
law
it helps the attackers just as much as
it helps the defenders and that's one
reason that I find this such a
fascinating area to work in in
particular current so current
cybersecurity approaches aren't working
very well and one of the reasons they
don't don't work very well is because
they don't handle what are called
zero-day attacks and an example of
zero-day attack was the Stuxnet Stuxnet
I don't know what the piece of malware
Stuxnet weapons Stuxnet whatever that
attacked the Iranian nuclear facilities
a couple of years ago and this this is
inter and it was very interesting well
as interesting for a lot of reasons and
I think I actually think it represents a
moment in history that will be
remembered in and written about but from
our point of view it was interesting
because Stuxnet took advantage of at
least five different so-called zero-day
attacks that is a tax that exploited
vulnerabilities that had never been
publicized or known about before and so
we'll be talking about those as we go
forward now you might be wondering I
mean everyone read about Stuxnet by the
way the other interesting thing about
Stuxnet is that it broke out of its
boundaries it's it's my it seems so
biological to me be my understanding is
I don't have personal knowledge of it
but but the general feeling is that the
people who wrote Stuxnet and who
launched it didn't intend I thought that
it would stay a secret they didn't think
that it would be discovered and see the
light of day and the way that it was
discovered is because it broke out it
broke out of these nuclear facilities
and it spread actually all around the
world and then the computer security
companies started noticing the strange
beast the strange piece of malware and
that was actually how the existence of
Stuxnet came to light so that's just one
example and you might be wondering how
bad is it there's a lot of scare stories
in the newspapers and I don't really
like this marketing of fear I think it's
you know it's a little bit hysterical
but the problem seems to me is certainly
very serious and it's it's also
notoriously hard to measure and so I'm
just going to give you a little bit of
data to give you an idea of the scope of
the problem so this is a study that was
published a couple of years ago and it
shows the number of new vulnerability so
these are vulnerabilities that could
lead to these zero-day attacks that were
discovered every year for the past ten
years and you can see there's just kind
of this continual rise up to 9,000 in
2010 and that's that's a lot of
vulnerabilities usually these
vulnerabilities come about through what
we in computer science call corner cases
that is highly unusual events or states
that the software gets into that that
leaves a little opening that someone who
understands the opening can write some
malware to break through and so this is
my last slide of scary statistics
symantec is a leading antivirus company
and they estimate the global cost this
is cybercrime so this is a little
different than the estimates I was
giving you yesterday but they estimate
the global cost to be one percent of the
u.s. GDP roughly that that's a lot of
criminal activity that's going on on the
internet and I was surprised to discover
the consumer reports good old consumer
reports that writes reviews of
refrigerators and washing machines and
cars do a study on malware and concluded
that in the year they did the study
one-third at least one-third of
households had experienced some kind of
malware event so that's you know that's
kind of like the old Harvard days look
to the left look to the right you know
one of you has been infected with
malware kind of thing and I have to say
I don't really I don't myself have any
way to assess these numbers but these
are just two kinds of studies that there
are lots of these studies published
and I don't know any one of them you
could poke holes in but I think if you
take them all together they're troubling
and they suggest that we need some new
approaches to this problem so as I
mentioned there are many different kinds
of attacks and there's more being
invented every day but I thought it
would be worth it to take take a few
minutes to consider some of the most
common forms of attack and just tell you
in a very high level how they work and
so the first one is this man in the
middle attack and the way this works is
as you know Alan works at Los Alamos
National Laboratory and Alan travels a
lot and Alan often travels to
conferences that are in very lovely
remote locations so we'll just assume
he's in the Bahamas or someplace like
that offshore and Alan's working on some
top-secret project for Los Alamos and he
wants to upload some of his results a
paper or something to his colleagues in
Los Alamos and so he does this through
the internet it goes you know through
some rowdy and there's some networking
and along the way I i'm actually not
working at the State Department this
year like they said yesterday I'm really
working for a foreign government let's
just suppose that and so I'm interested
in these top secrets top secret
documents that Alan's uploading and so I
have I managed to intercept let's say he
sends them over an unencrypted channel
or he thinks its an encrypted channel
but it's not really working very well
and I can actually intercept these
documents and then I can do I can do
several things I could make a copy of
them and then I could just send them on
and in that case it would be very
difficult for Los Alamos or for Alan to
realize that they've been compromised I
might also change them in some way or I
could just delete them in which case
probably everyone would know these are
these are the so-called
man-in-the-middle attacks and there's
many varieties of them and that's one
common form a second common form of
attack is the so-called botnet these
came along fairly recently
and I don't know if you can actually see
the picture but the idea is so instead
of instead of getting Alan's top secrets
now we're going to assume that I'm in
the business of delivering spam so
companies that want to send out spam
messages come to me and hire me to send
out spam messages to some number of
customers not their customers just some
number of computers and so the way I do
this is quite clever I first of all
devise some kind of an attack and I
using automated means break into sort of
compromise thousands of computers spread
all over the world and I don't actually
do anything to the computer so there's a
good chance they won't notice but I just
installed the so-called backdoor some
you know maybe I give myself a login
account that would be you know an easy
thing to think of there's other more
subtle things that you can do and so
then these these machines have become my
so-called BOTS and they just you keep on
if you're a bot you just keep using your
machine however when someone comes to me
and says now I'd like you to send out
spam to a hundred thousand people I say
no problem and i activate my tens of
thousands of bots all around the world
to send out these spam messages so this
has this has two advantages so there's
the little spam coming out of
everybody's computer this has two
advantages first of all I can have the
economies of scale I can send out way
more messages than I could send myself
using my own network connection and when
the network operators notice someone is
eventually going to notice a lot of spam
being sent from any single machine and
when that happens they're going to go to
the people who own the machines they're
not going to go to me so I can sort of
avoid detection so those those are two
common kinds of attack I didn't actually
tell you how I would how I would do the
break-in and so now we turn to what for
many many years I mean like maybe 30
years was the most common form
of attack and it's called the buffer
overflow and to understand this tack we
have to attack we have to do a little
review from last last night's lecture so
remember I told you about a program here
written in C or some high-level language
and how it and so the name of the
program this time is going to be victim
victim see it's going to get compiled
into assembly code and then into object
code and then it's going to be loaded
into the computer but this machine's
this program is a little more
complicated than the one I told you
about last night and it actually has it
sets aside when it gets loaded into the
memory it sets aside some a fixed amount
of space to hold some data that it's
expecting to read in when it's executed
so for example if it was a program that
processed email messages this little
buffer as it's called might be reserved
to store the text of the subject line
for the email message and so now what
happens is so the program gets loaded in
and executed so r e it reads in the
input for it takes the input for this
this email message and it takes the
subject line and now if the subject line
is longer has more characters than the
amount of space I've allocated in my
buffer this is just this is just the
main you know the main memory area of
the computer then the subject line is
going to actually spill over so if you
go back here it was only supposed the
buffer was only supposed to go this long
but this one email message has an
extremely long subject line and the
subject line doesn't just have
characters like the subjects you write
on your messages but the subject line
actually has little machine code
instructions and so when the buffer gets
overflowed it over writes this return
address pointer which tells the computer
which code to execute next when it's
finished executing this little program
and so if you if you're clever and do it
just right and know exactly how this
buffers laid out you can overwrite the
return address with an instruction that
says jump to some other
graham and if that program happens to be
the attackers program then then that
program gets executed with the
privileges of the original program ok so
this form of attack as I mentioned is
extremely common it's still extremely
common and it's actually been around
it's been known about since the early
1970s and I one thing I want to say is
that the reason this attack can work is
because of our early computer pioneers
who we revere so much John von Neumann
and his friends who invented the stored
program computer and so the thing that
happens in this attack is that the the
instructions get treated as data they
get red in as data when you process this
subject line and then they later get
executed and that is has given us a lot
of power in computing that's one reason
computers work so well but it has also
made computers vulnerable to this kind
of attack okay so these are just three
examples of how malware can gain control
of someone else's computer and generally
what happens is every time a new one of
these attacks is devised or a new
variation of is devised the computer
security community takes a look at it
they reverse-engineer it and then they
write they figure out invent a method
for coping with this new method of
attack and usually that new method
involves a so-called patch a bug fix
that gets distributed to all of the
customers and so the problem with this
is it doesn't scale it means for every
new kind of attack some human has to try
to understand it or maybe you could
imagine my automated software fixing it
automatically but even so it's it's not
a very scalable way to go about the
problem and before I start telling you
how I think we should attack the problem
I just want to tell you how computer
security thinks about security so they
there are three main properties that
that we care about for computers we care
about confidentiality and that's the
idea that my data if I want it to be
kept private should be kept private that
one
has been in the news a lot lately we
care about integrity so integrity would
have been if i took Alan's interfered
with Alan's transmission and changed the
data so we want the information that we
store to remain in the state that it was
when we stored it and not not be changed
out from under us and the final thing is
availability we want our computers and
our information to be available anytime
anytime we want to get to them and the
way that the computer security people
technically are typically go about
ensuring these three properties is by
writing rules or policies that they
expect either an organization or a piece
of software to obey and and these rules
or policies often get translated into
computer programs and then they get
loaded on to the computer and that's
supposed to enforce these these three
properties and of course what happens is
that the people don't do a very good job
of writing the rules and these corner
cases I talked about are problematic or
when the rules get translated into
software the programmers make mistakes
we talked last time about how common
that is or when the pro even if the
program is correct and the policies
correct if it gets loaded onto a
computer that has configuration problems
then all those policies might not might
not provide the guarantees that the
authors think and so this is kind of a
top-down heavy-handed way to go about
security and it it doesn't adjust very
well to changes and so one thing about
the computing world is it's always
changing there's always a new set of
libraries or a new piece of hardware and
those of you with smartphones there's
always a new app to be downloaded and so
it makes it very hard in that
environment to use this strategy so my
idea inspired largely by Alan was to
notice that the problem the immune
system solves for the body is
essentially the same problem that we
would like our site
security systems to solve for our
computers and in particular we immune
systems do detect novel pathogens
diseases you've never been exposed to
you might get sick but you eventually
can get rid of the infection under most
circumstances and when it notices one of
these infections when your immune system
does it can figure out how to get rid of
it and then it can get rid of it
automatically and usually it can get rid
of it in time to save your life before
it so much harm is done to your body
that that you can't survive and we what
we would like computer security to do is
the same thing we would like it to be
able to detect unauthorized use of
computers like I just told you about
malware and ideally we're really a long
ways from this is respond automatically
and clean up those infected computers on
their own and in fact I like to argue to
my computer security friends that
biology is the science of security it's
the true science of security in the
sense that organisms organisms have
evolved in this adversarial environment
for eons and they've had to contend with
with these attacks on themselves in
competition right from the beginning and
as a result they've evolved these just
amazingly elaborate clever complex
wonderful defense systems and a picture
of that is this an example is this
little immune cell right here that is
actually eating attacking and digesting
bacteria these these little things here
are bacteria and there's all kinds of
mechanisms like that one of the reasons
that I'm so interested in the immune
system is I think that or in or in
biological defense systems is because
they have they really have to cope with
very difficult challenges like
autoimmunity deception and mimicry so
here's just one example out of many but
it's one of my favorite examples this is
a mimicry attack so this is an ant right
here and the ant has been infected by a
parasite
and the parasite has the effect on the
aunt of making this part of its body
resemble the berries that the aunt is
living among amongst and so the effect
of that is that birds who come to eat
the berries might be confused and might
instead eat the aunt instead of the
berry and when they do that they'll
digest the eggs from the parasite and
fly to where they fly and when they
defecate the parasite eggs will be
spread to a new location and so I just I
thought that was a beautiful example
when I discovered it there there are
many other similar examples and and I
can tell you that our computer security
systems are a long ways from being able
to defend against this kind of attack
defense in depth is a popular concept in
cyber security people talk a lot about
defense in depth and what another reason
why I like biology is because it
exemplifies this strategy so biological
systems have layers of Defense they have
several defense and repair mechanisms
that exist inside every cell in your
body and I'm not going to talk about
those tonight but they're important and
they're always an area that I've
intended to go work on in my research
and haven't gotten to then there's quite
a few generic responses things like
homeostasis that keep the body the body
sort of monitors its own temperature pH
whatever its monitoring and makes little
little adjustments that are essentially
not noticeable by the organism that
keeps it within normal operating
tolerances and that's an important
that's an important mechanism and
there's several other related ones like
that it also uses this principle of
diversity which has been one of my
favorites so the idea of diversity my
interpretation of diversity is that it
confers population level robustness so
even if a particular individual is
susceptible to a disease or an attack of
some sort if there's diversity in the
population there's a good chance that
some of the individuals in the
population will survive the attack and
the
species will continue will live on and
then finally invertebrates there's this
beautiful defense system that seems to
have evolved especially to defend
against pathogens called the immune
system so here is a very simple
biological concept this concept of
diversity that is directly relevant to
our computer security problem and I like
to argue that our cyber infrastructure
infrastructure looks like this field of
corn it's essentially a monoculture in
the sense that most of us in the room
either have windows p species running
some flavor of windows or we now have
some of us have max running the mac OS
but we all use the same email readers
and the same software and so there's
this important homogeneity or uniformity
and that's good economically right
economies of scale aren't good that's
what let's computer companies make money
but it also provides on a source of
vulnerability to replicated kinds of
attacks and so just like I don't know a
little weevil I don't know what kind of
bugs attack corn but whatever kinds of
bugs kind of bug a tax corn once one bug
finds one ear of corn to eat then it's
and it's descendants can eat its way
through an entire field or state that's
planted with the same the same crop so
what we would like to do is take this
idea we were interested in taking this
idea and thinking about how to apply it
to computing so what we wanted to do was
have a way sorry this little microphone
is falling off okay we wanted to have a
way to be able to introduce diversity
into the computing infrastructure in a
way that wouldn't disturb the users but
would be highly likely to prevent
attacks or disrupt attacks and that's a
bit of a trick the idea is really simple
I thought it was a great idea and the
details turn out to be very intricate
and so I'm just going to give you one
example there's several different ways
that you might do this this particular
method
now actually made it into most
vendor-supplied operating systems so
most of you are running computers that
uses this so-called address space
randomization and so here's this little
cartoon of the main memory of the
computer from a few slides ago and let's
suppose when you the this right here is
all the different programs so I was only
telling you about one program there's
lots of other programs there's data and
it's all laid out in memory in this
certain way but now imagine that the
next time you run this program the the
memory is laid out in a completely
different way and so this little jump
instruction if it jumped you into this
code the first time and maybe that
attack succeeded the next time it's not
going to work or on the next computer
it's not going to work because the
attack code won't be located in the same
place and that's you know it's very
simple idea it's actually quite tricky
to implement and get it right but most
of the operating system companies have
become convinced that it's a good form
of Defense doesn't defend against
everything but it's a good good first
start now we're going to turn to a more
complex system the immune system and
especially with Ellen in the room I'm
only going to give you the fifty
thousand foot view cartoon view of how
the body defends itself against foreign
pathogens and by pathogens I mean
viruses bacteria and even parasites and
the thing about these particular
attackers is that once they have managed
to get inside your body or someone's
body some organism they typically start
replicating and the defense system then
has this time constraint that it has to
be able to notice that it's been
infected and eliminate the infection
before there's so many copies of it that
it has really prevented the body from
functioning and the way the immune
system does this this is my little
cartoon imagine this is one of these
attacking pathogens and it has gotten
into your body and it turns out
throughout your body you have a whole
set of detectors
whose primary job is to notice invading
invading foreign material and these
these cells called lymphocytes you may
have heard of T cells or B cells those
are both examples of them and they have
these receptor these unique receptors
each of them is is made a little bit
differently so they each have a
different pattern of receptors on their
surface and the idea is that
collectively the repertoire we'll have
enough enough diversity in it that at
least one of these detectors will
recognize that is it will be able to
bind be complementary and bind to the
pattern on the surface of the of the
infecting pathogen and once that happens
we call that binding process recognition
there's a little picture you know kind
of a picture next one step towards
reality of what it actually looks like
right there this is the detector and
this is the cell that's presenting some
of the antigen and once that once that
recognition event happens then there's a
whole cascade of events that takes place
depending on the particular kind of
infection it is that leads to the to the
pathogen being eliminated okay so this
this is a very clever very wonderful
system it's distributed throughout your
body and these cells for the most part
work autonomously they are independent
and decentralized there are also
molecules many classes of important
molecules that are secreted by these
cells we're not going to talk about
those tonight and one final two final
things to note it's important that these
receptors be fairly specific because if
they are general and can bind to all
kinds of different patterns then there's
a high probability that one of the
detectors one of the lymphocytes will
bind to some of the normal cells in the
body and lead to an autoimmune reaction
and we know autoimmunity happens it's
it's a common common reason for a common
explanation for many diseases and so
that's an indication that this
recognition
is what we would say in computer science
non-trivial ok so these recognizers have
to be specific and this problem of
distinguishing self from non-self is one
of the jobs immune system has to do the
other thing the immune system does is
when it's confronted with a pattern that
it doesn't recognize very well it
actually uses a process that looks a lot
like the genetic algorithm I talked
about yesterday it's looks a lot like
Darwinian evolution except without
crossover and in a few generations a few
days it evolves a set of detectors that
recognize that particular pattern very
very closely and that process is known
as affinity maturation and after that
happens some of those cells become
memory cells and those memory cells then
can respond much more aggressively if if
you're ever infected with the same the
same pathogen again and so that's why it
is that some diseases like measles you
can only get once so how do we actually
take that story and turn it into
computer security we use this thing that
I like to read this idea that I like to
refer to as the fundamental modeling
abstraction and Alan mentioned this work
this other modeling work that we've done
with Derek Smeath and some other
projects I've done and all those
projects use the same abstraction so the
idea is that these receptors on the
surfaces of cells that are just
molecules and we abstract away all the
detail about those molecules and we
represent them just as a string of zeros
and ones and so this is the detector
cell and then we might have the antigen
right here being presented by the
presenting cell and then this binding
operation that I talked about for cells
we can actually model that as just
matching string matching between two
different kinds of strings and in
computer science we have a lot of ideas
about how to do string matching one one
simple idea is just to use hamming
what's called hamming distance which is
just to walk down the string and count
up the positions at which the two
strings match and in this example
there's a pretty good match
so once we've made that abstraction then
we can build our little artificial
immune systems simply by generating sets
of bit strings and finding finding
patterns for them to match against and
we can actually mimic a lot of the
mechanisms of the adaptive immune system
okay so we had done a lot of modeling of
the immune system and then then actually
what happened is I got a job at the
University of New Mexico and a computer
science department and I needed to
convince them that I was a real computer
scientist so I had to solve a real
computer science problem and so I became
interested in this idea of building a
digital immune system and to do that
you'd have to have several things the
first thing you might want to have is
something like a nonspecific response
things like the skin and the mucous
membranes there are many sort of generic
defense mechanisms in your body and we
also in computers we have things like
firewalls and access control lists and
all kinds of basic permissions built in
even the basic virtual machine
multitasking model of computing can be
thought of as a kind of defense so those
are nonspecific defenses but the great
thing about the immune system is if that
fails it has this backup system called
the adaptive response and that's the one
that I just told you about and so if we
were going to do that for to attack
malware or detect malware we would have
to first of all figure out what we mean
by self and how to generate detectors
what the what the what the attack
patterns are going to you know what
space they live in you know are they
Network packets or are they patterns of
function calls or arguments of function
calls i mean there's a zillion things
that you might do and then we have to
think about how to process the patterns
and if we're going to do the whole thing
we have to have learning and memory so
actually the hardest part of this turned
out to be thinking about defining self
and if you think about it you need we
needed a definition of self that could
tolerate
all the things that people do to
computers like load in new software or
reboot they're operating you know reboot
into a totally different operating
system or edit files etc etc and it
turns out that system administrators do
the weirdest most anomalous things to
computers but they of course are the
ones we wanted to convince that this
approach would be a good idea so we had
to make sure that we thought about that
another problem in the computer is that
there's it's not so clear what the
periphery is you know it's pretty clear
in your body where the body stops and
the environment starts relatively clear
but in computers it's a little hard to
know when you're where your computer
start starts where your computer stops
and where the rest of the environment
takes off and especially that's true
cloud computing has just made that
problem even harder all right and then
in addition to having this response in
the long run we will also like to have
an automated response to be able to get
rid of the problem once we've detected
it and then finally I added this little
item because people always ask me about
it we would like to have a way to
protect the immune system itself so
before I tell you how to do that I just
want to come back to the zero-day
attacks and convince you that they're
there a real problem so in the past few
well really Stuxnet Stuxnet is a famous
one I talked about i mentioned five
different zero-day vulnerabilities that
took advantage of and in the end and
infected one and a half million hosts
around the world so that was a pretty
impressive infection especially
considering that most of it was
accidental and then this botnet the one
of the very famous botnets called
conficker infected over 450,000 machines
so this is really different than the old
days of these floppy floppy disks that
maybe went around an office or something
this this is a serious problem and it
turns out actually that there's a market
there's now a market for the zero-day
attacks and although I have not verified
this personally some of the accounts
I've read
that some of these attacks sell for as
much as five hundred thousand dollars
that's a lot of money and the thing I
just want to remind you that the thing
about these zero-day attacks is that
virus scanners and I don't know if
you've heard of snort there's a variety
of these kind of signature based
intrusion detection systems will talk
about that on the next slide they are
not going to be very helpful against the
zero-day attacks I they might every now
and then accidentally catch one but in
general if you have this theory that
you're going to wait to see the attack
and then you're going to engineer the
defense against it you're always going
to be one step behind so after virus
these virus products have been around
for a while people had the idea of an
intrusion detection system because
that's really what the immune system is
right it detects intrusions to the body
and so the way this looks in computers
you have a computer here this is your
monitored system that we're going to
monitor for intrusions and you have to
identify some data stream this is the
thing that I was saying is like defining
self so you have some data stream that
an external monitor can watch and
collect data data on so that that's
where showing that hear that whatever
the data that we're watching we'll talk
more about that in a second comes off
the machine we take a look at it and
then we feed it into a model and the
model is supposed to tell us whether or
not that's normal behavior or whether
there's an attack going on so at all
everything rests on the model to look at
the data and decide whether or not the
system is under attack and normally the
way these intrusion detection systems
work is that a programmer over here
writes down these so-called signatures
for each attack and that's a little bit
like having an immune system that
protects you only against diseases that
you've been vaccinated against so it's
good especially serious diseases that we
know about but it leaves you pretty
vulnerable to the new one that's just
showing up in that Egyptian to bet so
the strategy the immune system uses is
to have this model be built kind of
built automatically and to have it have
it work on the principle of anomaly
detection that is it's going to notice
what the normal patterns are for the
body and when something unusual comes
along it's going to treat that as
dangerous or foreign and it's going to
attack it and so that's the idea that we
want to use and actually I should say
the immune system also uses these
signatures because once it's learned
what these patterns are it saves them
and then it can bring them out so it
actually has both of those abilities so
I'm talking about this example even
though it's kind of old by now this is a
system that my student Stephen Hoffmeyer
built for his dissertation because it
has the most amount of immunology in it
and it was a network it was it was a
kind of a network intrusion detection
system it was very innovative at the
time the idea was you can tell that it
was a little while ago because you can
see the pictures of the computers are
kind of old anyway so we have a bunch of
computers on what's called a local area
network and the local area network is
outlined herein it's supposed to be
yellow looks kind of mustard colored on
the screen and the computers as you know
talk to each other and it turns out that
most computers talk to very few other
computers over very limited what are
called services using a very small
number of programs so a service might be
HTTP which handles your web requests or
stmp which handles your email but a very
few number of those programs that that
we want to communicate with and the
computers deal it's let's see what else
do I have to talk about the computer the
computer each computer has a unique
address this is the so-called IP address
and it's the series it's represented as
this series of four numbers which is
kind of a convention and the idea is
that we're going to make a little immune
system a detector set for each one of
these computers and actually we're not
going to make it we're going to let the
computer build its own and evolve its
own in the network of the
of the of in the environment of the
network packets okay and by the way
local computers on a local area networks
some of them at least tend to talk to
outside computers so we have to be able
to handle those as well and the attacks
could come either from inside the
network or from outside so our
definition of self in this example what
one of these detectors is going to
represent is one connection between a
computer computer a and computer be and
so the bits the first set of the bits is
going to code for the source IP address
where the connection is coming from and
the next set of bits is going to code
for the destination the address where
the packet where the connection is going
to be going to and the third one is
going to be the port the kind of program
that which kind of program we want the
receiving computer to use to understand
the network communication I'm
simplifying by the way a lot of the
details about modern networking I
apologize okay so how did how did this
actually work well going back to our
fundamental modeling in abstraction what
we would do is generate these detectors
we would actually generate them each
computer would generate them randomly
and then we would let them live in the
environment of the of the network and it
would match all the connections incoming
connections and so this is a picture of
a it was actually only the syn packets
only the packets that initiated that can
the connection that we looked at and
that made it possible for us to do this
in real time which was important and
then we had to define a matching rule
let's just assume the one we used is
very close to this Hamming distance
thing I'd said before so we have our
detector which we generated randomly we
have some network connections and then
we just go through and count up the ones
and zeros and see how many positions it
matches at and then we define a
threshold and I think in these
experiments if i'm remembering we had
about a hundred detectors on each
computer and these these detectors it
turned out through a few tricks were 49
bits long
and we required we define a match to be
about it was a roughly two-thirds of the
bits had to match in order in order for
us to declare that that was an anomalous
packet okay any questions about that
okay good cuz I can't really see you
anyway I kind of like this you know when
I lecture when I lecture in class of
course people get their hand up all the
time and interrupt me this is kind of
nice i don't have to i can just keep
rolling okay so um so i just want to
focus on on one of these detectors but
just remember that we had a whole set of
computers and each computer had a
collection of about a hundred of these
detectors and we would we would randomly
generate them this is not unlike this is
taking a few liberties but a reasonable
approximation of how the immune system
generates its receptors and then we
would let let the detector live as i
said in the environment of the of the
network and just watch the network
traffic and if it matched anything in
the first two days was our default value
than we would kill it off and so the
idea was we wanted to train these
detectors we wanted a set of detectors
that was not autoimmune that didn't
match with self and so we were training
it on the normal behavior of the network
and it turns out that that this little
operation this was the one that Alan
suggested suggested is very close to an
immune system mechanism known as
negative selection and if it did match
anything we killed it off and then we
just right away generated another one
yeah
I'm sorry I let me just take that at the
end okay I think that's more technical
than most most of the audience okay but
I will be happy to answer that question
I just couldn't I couldn't quite hear it
and I think it's going to I have to
explain too much networking to everyone
else too to answer that so let me just
plunge on i but i but one answer is we
actually look at both incoming and
outgoing connections I don't know if
that's the question you were asking but
that that's one one additional piece of
information okay so if it actually
survived this negative selection then we
declared it to be a mature so-called
naive detector and we then put it to
work and said okay go forth and detect
anomalous network traffic and if it
failed at that job if it just didn't
let's say it doesn't match anything then
after a longer period of time normally
about a week we would kill it off and
that is also not unlike what the immune
system does so my student actually built
this system first and tried it out and
ran it and it turned out to have a huge
number of what we call false positives a
lot of autoimmune reactions and because
he wanted to graduate with his PhD he
immediately got very clever about how to
fix it up and he started inventing these
new mechanisms and so one mechanism was
to say uh well these naive detectors
they're too quick off there too quick
off the mark they have a hair trigger
and they're reacting to everything so
what we'll do is give it a threshold and
say that it has to match a fixed number
say ten of these anomalous packets
within some period of time so we call
that the activation threshold and if it
would if it exceeded this activation
activation threshold then we would say
the detector had been activated and
again this was the next little step and
then we were going to send an alarm to
the system administrator that still gave
us too many false positives it was a
little better but we were now in this
home of research and I just really want
to communicate what it's like when
you're we try out these things this is
exploratory research we actually don't
know if it's going to work or not ahead
of time we kind of have this you know it
should work it works for the immune
system dang it if I just push hard
enough I should be able to get it to
work but it's there's always that big
question mark hanging out there and when
it's a student who's doing his
dissertation or her dissertation there's
even more drama okay so then we had it
so then Steve went away and scratches
had he added a few more mechanisms and
the next mechanism is what in immunology
is called a second signal and so he
required the in our case it was a system
administrator to confirm that this was
actually a an anomalous event we you
could imagine other automated ways of
doing it and in the immune system it's a
helper t-cell that does that and so when
we put all that together we actually got
great result results that was finally
enough and in the end it was able to
detect all of the standard network
attacks of the day we deployed it in our
department on our local area network for
a while and it detected many interesting
things many unusual behaviors including
one of the teaching assistants who was
port scanning his professors machine and
it was sort of humorous who the
professor was that was part of the humor
okay so what we learned out of all this
was that autoimmunity is important and
requires a lot of mechanisms and it was
actually surprising how many of these
mechanisms that had correlations in the
immune system how many of those were
really required and we also this cross
reactivity idea that it was sort of
sloppy matching with a threshold that
was one of the things one of the tricks
that made it possible for us to run this
thing in in real time with a reasonable
number of a tech ters okay so this work
and some of the other projects we did
then had an amazing amount of impact
more than I could have ever imagined on
and it turned out that a system that one
of my other system students built that
he got working sooner than this one was
the first is it's generally credited as
being the first practical anomaly
intrusion detection system that one
actually used system calls my students
Steve the one who did that work when I
uninformed a company that made a product
that sold an intrusion detection system
called primary response that worked very
well didn't we couldn't out-compete
Cisco sort of on market share but
technically it was better and then we
also investigated a lot of these
graduated and automated responses and
one of our ideas there got picked up by
hewlett-packard hewlett-packard and
turned into a product they called their
virus throttle we we did a lot of other
things and I can see I'm running a
little short of time so I'm not going to
tell you about this other one but
another project we did we took this idea
of negative representations and you
could think about these negative
detectors as providing sort of a
negative image of the pattern of self
it's kind of a negative representation
of self not unlike this famous earned
picture that shows you know where you
have the figure ground issue and so we
we use that idea to develop some very
unusual privacy-preserving data
representation techniques that work has
not had as big an impact but maybe it
will now that we've learned all about
the problems with the regular encryption
I don't know okay so you might be
wondering if you're skeptical do we
really need this immunological
perspective because a lot of these ideas
these anomaly intrusion detection
systems are now out there in computer
security no one calls them immunology we
have address space randomization no one
calls that immunology so why do we need
them and so I want to close just this is
the last little part of this talk by
telling you this fun anecdote I have a
collaborator named Jed Crandall down at
UNM he's quite a bit younger than I am
in fact I hired him when I was chair so
you can imagine I you know I was
technically his boss and he he's a
hardcore system security
you know operating systems networking
our computer architecture kind of guy
and we go out to lunch a lot we're both
interested in computer security and he
was always very polite probably because
I was his chair and listen to you know
listening to me talking about all this
you know bio babble and he never really
believed it very much he he was very
very skeptical and but but we kept
talking and we've always liked each
other anyway so he came to me one day
about a year ago maybe a little more now
and he was kind of shuffling you know
and hunched over and he said I really
would like to talk to you about
immunology and I I'm immediately
thinking okay he or someone in his
family has some terrible disease or
something you know Chad's never going to
be interested in immunology and he says
well I'm not liking it very much but I'm
doing this work on on internet
censorship and I think what I'm seeing I
can only explain by these ideas about
immunology and so I just want to close
out the talk by telling you a little bit
about how that goes so what he's
interested in he studies China a lot but
the ideas are pretty general and he's
interested in what information is China
blocking where is it being blocked it's
not always being blocked on the outside
border like people might some of the
articles might tell you when is it being
blocked and how is it being blocked and
so his research is really to define the
develop the software tools to let him
answer these questions sitting in New
Mexico not being inside China and
they're very sophisticated and fancy and
so he's done a number of case studies
he's looked at at local sites local
networks to see what's being censored by
the way Wendy's does a lot more
censoring and has their networks shut
down a lot more than our Albuquerque
Police Department that was an
interesting discovery and he looked he's
looked at some of the Chinese chat
programs and finally he's looked at the
Chinese version of Twitter which is
called Weibo and it turns out there's
several layers of these attacks
the the first layer these are used in
the chat programs and they're also used
in Weibo are these keyword black lists
and so we spent a lot of effort trying
to figure out what keywords were on the
blacklist coming up with funny examples
and and all of that like tanaman square
might be might be a blacklisted kind of
the Chinese characters for that and so
that's kind of the first level and then
the people who wanted to communicate so
let's say about Tiananmen Square without
being censored they figured out a way
around it which Jed refers to as a
neologism and so you can see here so
Wakhan is going to feature in this next
little story we tell who con is a place
in China and the character from Luke on
looks like this Jed by the way while
he's been an assistant professor has
also been taking Chinese classes and
this is another character this word i'm
not sure what it means but the character
looks a lot like Luke on so when people
wanted to talk about woke on they would
just send this character and everyone
that was sort of in the in group would
know that it was referring to this sort
of unmentionable unmentionable word so
that's sort of the second the second
level and then level three are these
dynamics and it seems like it's a very
very adaptive process and I'm not going
to go through this in gory detail but
this a couple years ago in Wuhan which
is a province or a town someplace in
China there were a lot of protests and
people were writing blogging about these
protests and so the this is time and
this is a log scale to that let you know
how many of these posts these Weibo
posts mentioned this particular term
rukon and so you can see that it goes
way up when these protests first started
then it kind of dies off and bounces
around and then at some point in here on
someone was killed the the authorities
killed someone and that made a lot of
news and got a lot of people upset and
so you could see that it spiked up even
more and remember this is a logarithmic
scale
and then it suddenly plunges but it
turns out that this term right here
which is shown in red the frequency of
that took off right about then and so
what happened is when the Chinese
started censoring the one term the
people who were communicating use it the
use the other and so one of the well
there's several things to notice it's
threshold-based right they were
tolerating a little bit of this talk for
quite a while but eventually it crossed
a threshold where the sensors decided to
shut it down and then there was a
peaceful resolution to this particular
situation that happened right in here
and so after that the Chinese quit
censoring and so it went up again and
then it kind of died off and so this
illustrates Jebb thinks jed thinks this
is very kind of a typical example and
that the censorship is first of all so
how did he do this I forgot to tell you
that how did he do this well all the way
Bo posts are public right just like
Twitter so he downloaded them and he has
a large collection of these things that
go that use up a lot of spade a lot of
storage space in our department but I'm
not sure anymore so I don't have to
worry about that problem anyway so uh
what he what he is concluded about
censorship and with many of these little
stories and observations is that first
of all it's decentralized it's not all
happening in one place and that it's
multi-level its defense in depth first
there's keyword blocking then these neo
geo legends get used and it's adaptive
so when a new thing comes along and
suddenly become sensitive like this Luke
on the the system can respond and event
takes a little while and it tolerates a
little bit of noise but eventually it
blocks it and so there's this idea of a
graduated response which we've used a
lot in our own work its robust to small
amounts of noise and when I was talking
to him about it I said well obviously
the next step is the sensors are going
to figure out a
to detect these neologisms and then
we'll have cross-reactive defenses just
like the immune system so I think that's
a nice a nice illustration and you might
be thinking well this isn't so good
because we think censorship is bad but
just remember it all depends on which
side of the fence you're on and so if
you're the Chinese this is just a way of
defending keeping out this bit these bad
posts and so it's interesting to me that
it's a it's evolved to this this kind of
complex state that looks a lot a lot
like the immune system okay so to sum up
malware attacks are prevalent and they
use a lot of very clever tricks some of
which I showed you we can use biology in
various ways we can use diversity to get
system level robustness and disrupt
these replicated forms of attacks and
that technology is out there at least
one form of it is out there there are
many other forms we could do and we can
use immunology both for security like in
this intrusion detection sent sense and
what I didn't get to tell you about is
also for protecting privacy using these
negative representations so that's good
news that's the good news however if we
go back to this picture that we looked
at last night we might have a lot of
questions like where should we even if
we have these mechanisms these cute
technical mechanisms where should we
deploy them what impact will they have
you know if I just turn one of these
things loose it was okay when Steve did
his dissertation just to do it in my
little department we were all friends
and everyone let us do it but if we were
going to turn it loose in this world
what impact would it have and what are
the actual economic incentives that are
driving both the malware and the
defense's and it's very very costly to
do sort of empirical studies you know
just try something out in this world and
so the topic of next the next lecture
tomorrow is going to be how to build
believable models that we can use to
start thinking about these these human
factors that I've really completely
ignored so far
so things like incentives humans
incentives economics and increasingly
over the past just the past couple of
years as we've seen politics are
starting to really come into this world
and have a large influence on the kinds
of solutions that that are possible and
and considered usable and so we will
talk about that next time first we'll
talk about modeling and then we'll talk
about 11 particular kind of model that
uses ideas from biology and then we'll
come back to this question of policies
and incentives and talk about how we can
start thinking about those so thank you
very much
okay okay would you like to hear the
more technical or be sure I just yep
penalty ya madda metas I don't care
security okay I don't care pick your
pick your favorite and then we'll see
what everyone else wants is what do you
feel is the future of generic attack
techniques even with return oriented
programming in in this world where
modern memory allocators even custom we
have heap chunk headers protected by
some form of canary and all classic
stack-based buffer overflows are
similarly protected with the
proliferation of various pointerly
techniques to break a SLR for other
right yes address space allegan I got it
with that she took out mag I got the
question let me can I answer it let me
just a sort of translate for the
audience so this this account that I
gave up the buffer overflow attack since
that time well first of all there been a
number of different methods she
mentioned Canaries there's a lot of
other methods that have been developed
to to defeat that particular kind of
attack but that same strategy these
return to lib sea and return oriented
programming attacks are there sort of
that they're basically the same idea
they're just more sophisticated forms of
it and in fact on there are other kinds
of diversification like instruction set
randomization that would be helpful
against those kinds of attack but I
would I would classify that particular
world of these little memory memory
overflow kinds of attacks as having its
own little arms race with the defenders
I see so you think that we should start
formally considering proliferation of
return yeah well actually what I really
think is that automatically evolving
attacks is not going to be that hard as
I oh I think she was just asking
you whether I think these more
sophisticated forms of memory memory
attacks are enough of a problem that we
need we really need to think about them
widely and I think yes they are and I
think we're shutting down I mean it's
it's a lot harder now to do those
attacks then than it was when they were
invented but i also think in addition to
that that it's not going to be that long
before we start seeing more kinds of
automatically evolving forms of attack
the ANS the NSA uh I was wondering how
long it was going to take to get to the
NSA well the NSA has done a lot of
things and and I will tell you that most
of the things they've done I don't know
about and I the most recent thing that
came out that I think you might be
referring to are these attacks on
encryption systems and this these
particular attacks that I was talking
about today are not a tax on encryption
system on encryption mechanisms so
that's a different set of problems v and
yes the crypto community I'm sure is
very busy right now responding to that
information
if the port doesn't detect the source
but no the way the way it happens is if
I'm a computer and I want to talk to you
on my computer sends to you one of these
so-called syn packets and and it can it
basically says I would like to talk to
you I'd like to establish a connection
at least for this particular this
particular way of communicating and then
if you get my packet and your computer's
up and your that port is active and all
those things then you send back to me an
acknowledgement message and then
actually I send you a third message so
it's a it's a three-way handshake and
then after that the connection is
considered to be open and the packets
just flow okay back
um well that's people say they do a lot
I mean that's that's the common belief
and I guess they have an advantage in
the sense that they can they can buy
computers and they can buy software and
they can study they can study the system
that they want to attack and and they
also have an advantage in the sense that
as a defender I have to defend against
all possible attacks ideally right
because because I don't know which which
kind of attack might might come to me
but as an attacker you only have to find
one one weakness in my computer so in
that sense I think the problem is easier
for the attacker the biological analogy
is quite compelling for natural kinds of
phenomena but what we see in computer
systems are engineered attacks and
biological systems are not very good
against engineered attacks like sarin
gas or various poisons has come about so
how we defend against those engineered
attacks using biological ideas okay so
bob is a friend of mine so I'm going to
make a really snotty reply okay on that
I claim is the argument for intelligent
design and that's basically saying that
the intelligent adversary the engineer
is smarter than nature and I just don't
buy it in the case of I mean in the
general situation I don't buy it in the
case of the that you mentioned those
molecular kinds of attacks the immune
system is not not evolved to handle at
all it's not interested in them and
maybe Alan would want to say something
more about that but I I think most
engineers think that they're smarter
than nature and I in the you know in the
long run I just I don't think that's
true and when we have computer security
systems that are as sophisticated
as our biological defense systems which
even though they can't defend against
sarin gas they can defend against a lot
of other stuff then I'll sit down and
have that conversation with you again
look like i said bob is a friend of mine
so if one looks over evolutionary timing
biological systems have involved to
protect against all sorts of toxins may
take them tens of thousands of years to
do evolved mechanisms doing that so what
you're asking you know is something
that's very immediate but if we look at
all sorts of plant bacterial toxins we
immune system has discovered ways of
fighting against them
that's a great question did everyone
hear it ok so the observation is that
over time biological systems have
actually incorporated a lot of foreign
material things that used to be enemies
are now you know incorporated into our
genomes and we we use them for good
purpose and so is there anything like
that in the computer security immunology
analogy and i would like to think about
that question I can't think of an
example right off the bat on it's an
intriguing suggestion and I I can't
think of an example right off the bat
that speaks to that particular question
but it's getting late let's just do one
more question and then I guess anyone
who has burning questions come up and
talk to Stephanie separately this last
one back here let's be glad we
did did everyone hear that okay I I
think there were some knows there okay
so I i will reformulate the question in
a way that i can answer it so the idea
is that in sociology and i think most of
us would agree that our self isn't just
the physical stuff that the immune
system is evolved to protect against but
there's social norms and i don't know a
bunch of other things that he mentioned
that sort of are at a different level
and not really addressed by the natural
immune system and do i have any ideas
about how that's going to work in
software and i guess i would say that
that's part of why i'm spending this
year in Washington because I've realized
that we have these these technical
solutions that by themselves are
actually very good but there's major
questions about how to get them
distributed there's incentive questions
why do individual users feel so little
incentive to secure their own computers
they well they don't know how to but a
lot of it is they're not willing to
spend money to do that so why is it and
what is it that we would have to change
so I think those questions are all
coming up and I don't know how well
addressed they will be by the natural
immune system analogy i think that's
that's a good observation and it's
somewhat what I'm going to try to
address tomorrow you want to drive up
from Albuquerque again okay well let's
all thank Stephanie for a marvelous talk
you
