- Hi, thank you all for coming.
And it's especially nice to
see some non MFA photo people
who have joined us today.
For those of you who don't know me,
my name is Adam Bell, and
I'm an academic advisor
and faculty member in
the MFA photo department.
And we are very excited
to have Dr. Lev Manovich
here to give a talk.
As a bit of history,
I'm especially excited
because I remember when I was
a student not too long ago,
or actually maybe a little bit long ago,
back starting in my first year in 2001,
I remember buying and reading
Lev Manovich's famous book
The Language of New Media
and being amazed by it,
and here was this brilliant book,
so it's really great to have him here now.
For those of you who don't
know a little bit about Lev,
I'm gonna do a sort of abbreviated bio
which I've taken from his website,
which I'll just go through quickly here.
Lev Manovich is an
artist, computer animator,
designer, and programmer,
as well as an author
of numerous books,
including his most recent
which is Software Takes Command,
Soft Cinema: Navigating the Database,
and the Language of New Media.
He is currently professor
at the graduate center CUNY,
and the director of the
Software Studies Initiative
that works on the analysis and
visualization of big cultural data.
In 2013, he appeared in
the list of 25 people
shaping the future of design.
And in 2004, he was included in a list
of the 50 most interesting
people building the future.
In addition to being a
former Guggenheim Fellow
he has also received grants
from the Andrew Mellon
Foundation, The National
Science Foundation,
and the National Endowment for the Arts.
For those of you who want a closer look
at some of his work, I would encourage you
to check out his website, which
I'm sure he's going to show
today, which he's worked
on called On Broadway,
And also to check out the
Public Eye Photography show
at the New York Public Library
which is up now and I think,
does it run until next March?
Until January, okay.
Which is a fabulous show and I highly
encourage you to check it out.
It is a great honor to have Lev here,
and this will be a fantastic lecture.
(applauding)
- Well, thank you Adam,
and thank you guys so
much for organizing this.
We hope it maybe it won't
be my last engagement.
We're talking about maybe doing
a workshop with some students,
so hopefully you'll still
like me after this first date.
Now I know that some
students were actually
required to come here so please
don't hate me after this.
And if you hate me, I'll buy you beer.
Today I'll be talking
and actually showing you
a range of projects we
have done in our lab
which try to look at massive
image collections using
techniques of data visualization
and also contribution.
And I'm specifically going
to focus on on projects
where we look at
photography via Instagram.
But before I show those projects,
we are going to make a kind
of more general statement.
I think last year President
Obama said that to be
literate in our society you
have to know how to program.
I mean, now, I'm sure
you have other people
you maybe admire more or respect more.
But you know when the
president of the United States
says everybody has to
program, I mean I was
very happy to hear that
because those of us
who have been involved in digital arts,
I've been involved for
about 50 years, of course,
with I've been dreaming
about the moment where people
will actually say, "Hey, it's
not enough for me to know
"how to press a button on my
laptop, or make a power point,
"I should also know
something about the code."
Because the code is to modern society
is what I would say electric
engine of the trains
and other technological
forces were to that society.
Everything has a code.
Well, we know it.
Even President Obama knows
it, so it's not really news.
But what you guys may not
have realized is that in fact
if you think about visual
culture, history of media,
turn of the century we've
got photography, fax,
we've got television invented in 1870.
And of course all these things
become repeat in 20th century
you know radio and television,
video, internet, and so on.
But I think that maybe
five to 10 years ago,
in fact our society has
entered a new stage.
Or, let's say our media culture
has entered a new stage.
And amazingly, nobody
has written about it yet
as far as I know, so I
don't have a name for it,
but maybe we can call it something like
application media in cultural analytics.
So basically a significant
number of cultural interfaces
that the companies put out for
us to in our culture depend
on computational analysis
of massive amounts of media.
So while I'll be using
some of these techniques
is a way to do a type of history
or archeological present.
I think you want to be
curious, and I think
you want to learn about
these techniques just because
if you want to be a literate photographer
or literate artist in
2015, you should know how
mass visual culture
works, you should know how
cultural photography
works, and I'm telling you,
not all of it, but to a
significant extent it works on,
not just on code, not just
on my sequel, python, code,
it works on the particular techniques
of analyzing massive amounts of media.
Think about Google search,
how does Google search works?
Well, Google sends robots, we go online,
we travel from one webpage
to another webpage,
Which yields about 40 to 50 webpages today
according to best estimates,
it extracts all the
content from every webpage.
It analyzes all this content.
Whatever it is, it breaks it down.
If it's images, shows you images,
what's on the database in this case?
And then when you search for something,
it will return pages which
you think are most relevant,
or in the case of images, or
media to be images of video.
So search, which is now our
interface to information
as opposed to say a library catalog,
depends on massive processing
of cultural content,
such as webpages and other stuff online.
Recommendations, so if I go
to Amazon and that leads,
they are going to recommend
me some movies to see,
or some books to buy, or baby diapers,
or maybe some super expensive
lens for my camera or whatever
but how does Amazon knows what I like?
Well, it looks at what I'm looking at,
it looks what I buy, it
compares me to the same
histories of billions of
billions of other user sessions.
And then it does a calculation.
So in this case, my calculations of modern
media, such as photograph over
pages, but the calculations
are done on my cultural
preferences, my buying history.
We'll go to maybe one of the more popular
kind of newer news website,
such as Mashable.com.
I was talking to a guy that
was a chief data scientist.
And he says that, so
he built an algorithm,
which basically recommends to journalists
every morning what we should write about.
Based on processing billions of numbers,
what people look at before,
what people looked at last year.
But I think what's even more amazingly,
when he told this I was
just totally shocked.
They have algorithm which is in real time
adjust the position articles on the page.
So it analyzes the data,
when people are visiting
its website, using some
old data, and it basically
uses this, automatically
just positions images.
Like, you look at this,
it looks very innocent.
But it's a kind of, like
an alien from a new era.
Except a new era we're really living in.
Most of you who are
obsessing about becoming
famous photographer because
you're going to take
this one amazing photograph.
I'm not against it, but you
have to be a real genius.
Because so many great people
took so many great photographs.
But if you want to get
the algorithms and codes
and affect processing
of visual information
affected by the massive image collections,
and then combined it with artistic ideas
and intellectual ideas,
maybe your photography,
you can do some really bad
project and become famous
because this is completely new.
At this point, industry is decades away,
while we're still peeking
and looking at single images,
we've just been processing
billions of images,
billions of data points
to create this graphic
experiences for us, so how
can we take these techniques
and actually use them when
working humanistically
to do some interesting
feats, to do art, and maybe
to also understand patterns
in history of photography.
So I'm going to start with a
recent project we have done
that was an invitation from
MOMA, is in modern art.
We've been working for a few years
on a big, big project,
which is now exhibited
called Modern Photographs from
Thomas Walther Collection.
It is an amazing exhibition, so go see it
if you can afford $25, but you
can get artist's membership
actually and go lots of times for free.
And as apart of this exhibit,
we commissioned people
to write essays, so we invited my lab
to look at the photographs
of this collection
and see if we can use
computational and visual
techniques to maybe say
something interesting.
So I said one of my students,
he was a research fellow
last summer, 2013, he
was really nice to them
And after awhile we gave
him a whole digitized
photo collection of Walther.
So I don't know if it's
all, or really all,
nobody knows, but 21,000 photographs.
So how does 21,000 photographs
from MOMA looks like?
I'll show you.
This is a 70 page power
point we presented to them
of our findings, and we eventually asked
if we could use this broad data.
You can look at this
online, but I'm just showing
all the stuff, you can make into a print,
and we also have a side agreement
that we're not allowed to show
it to anybody or publish it.
But here it is, but you guys
don't tell them alright?
Anyway.
So just one more second please.
Okay, here we go, yeah, this is the one.
This is one of many
revisions we have done.
What it is, it's 21,000
photographs from MOMA collection
simply organized by year.
And of course it reveals something,
and we suspect that MOMA
collection, because for decades
MOMA was the first museum
to collect photography.
MOMA was really important
in making photography
part of our art of history.
But you may expect that maybe very strong,
different, more avant garde,
but it's really really strong.
So basically this is, in a
century we have something,
and then between 1920 and 1940,
look at how huge their collection.
And then it was really
little from the 50's,
and then there's a little bit
more maybe in the 60's or 70's
and then there's all
this nothing after that.
You can see why it seems non-published
visualizations of websites because
whenever everybody know how
particular recollections are.
But the reason I'm showing
it as the first thing,
yes of course you can present
this information as a catalog.
But that looks very abstract.
So in our lab we develop simple software,
it's opensource software,
we've been giving it
away fro free since 2010,
you can actually make
these revelations yourself, so my ideas is
to create visualizations
of image collection
which would not just show
you patterns in collection
using points, bars, et cetera, and so on.
All these kind of data relationship porn
as something people call it, but in fact,
maybe visualization out to
images in the collection.
So if I start zooming in,
as long as the computer
can keep up with me, eventually
I'm going to actually see
the actual images that these
calculations are made of them.
And then of course these bars now become
very very meaningful because
of course what you see is that
these images, which we can
dominate in the MOMA collection
is not just any particular
images, but the images
of very particular modernist
avant garde photography,
very abstract, black and
white, some kind of fragmented
body parts, lots of
different visualization.
Maybe you notice some type
of ostracized visualization.
And so on and so forth.
In fact, the collection
is dominated by a very
particular type of photography.
So it in no way represents the history of
photography as it actually happened,
it doesn't represent
the natural photography.
In this case, we can actually control it.
Now, if you just saw,
once you start creating
this visualizations, we
actually want to show
massive image collections on one screen.
Things like laptops or
these awful, awful devices,
the mobile phones are basically awful
because they prevent you from thinking.
So the last 15 years were a kind of waste.
In a certain sense, in terms of artistic
digital visualization because
everybody trying to make
those little apps, and this
is Atari, like the 70's.
And if I only would stare at
this awful device, I would
never think about, let's
visualize the MOMA collection.
I was really lucky,
until I came to New York,
where you wouldn't even find
the space to put this wall.
Well, maybe you guys can find the space,
but for the purpose of art,
the visual art for 20 years.
And I involved with a research institute
called California Institute
for Telecommunications
and Information where people
basically are devising
of information structures
for the next generation.
So it's devising next
generation's internet,
next generation's computing,
where CPU will be here,
and the graphics card will be in the back,
the screen will be in Mexico,
so the computer's distributed.
Internet, which is not
using normal cables,
it's using optical fiber,
so it's super fast.
And also designing the
next generation's place.
Because we are saying correctly
the world is connected.
We're all talking to each
other, it's all uploaded,
written, liking Instagram photos.
And we're producing massive
amounts of cultural content,
and cultural communication,
so this device is totally
devoted, you're basically
looking at the world
through a little peephole.
I mean, yes, it's amazing in one ways.
As a communication device, it's amazing,
as a device to think
about art, it's like...
(blowing air)
I would break it, but I just bought it.
But maybe if you invite to lecture again,
I'll break my phone, I'll
just kind of break it.
So these devices are much better.
This is the kind of visualization walls
I was lucky to work with in San Diego,
sort of one of many to be built.
So this particular wall
consists from dozens of
large, four to six
monitors, so it's like, then
a bunch of PCs with
high end graphics card,
the kind of things you
use to play video games.
We hired it so it's a little
visual separate computer.
It's like, along the
visual image collection,
and you'll see the second
sorted, it'll do exploration.
The kind of things that
maybe photographers
or photo editors in
magazines, like Life or Vogue
or whatever were doing for decades.
You put your photographs
on the wall, and you decide
which photo shoot is going to
go where, what's editorial,
what are you going to select?
This is in fact a very
I think common practice
in a photo journalism community.
But what if you can do it with computers?
What if you can actually do
it with billions of images.
Let me show you a video
which was shot when we
spent a few months in developing
the software for this wall.
It's kind of old, it's 2009.
But it communicates the idea.
- We're here at the HIPerSpace
Wall at Calit2, and we've
loaded an image set of the
works of painter Mark Rothko.
Using software developed
by the software studies
initiative and the HIPerSpace Wall team,
we're gonna be exploring
this set of paintings
using cultural analytic
techniques, turning the paintings
into sets of data that can be graphed,
and turning those graphs into
collections of paintings.
First, let's take these
images and let's move them
over onto part of the wall's surface,
and then load some graphs on
the other part of the surface.
These graphs all run over the
years of Mark Rothko's career
from left to right, but their
heights are indicated by
different features of
the images themselves.
Texture, brightness, number
of shapes, saturation.
And we can use them to explore
trends in this painter's life and work.
So let's organize this
tile set that we have
by one of these dimensions of data.
We can sort through different axes
looking at something as simple
as just the sequence of files
which we can view at different sizes,
all the way down to a series
of dots, and we can size
all the way up to high
resolution textural images.
I'm gonna turn the size
on this down slightly
and then I'm gonna add
a transparency effect.
You can actually see the original dot data
in the midst of the color cloud,
and by mousing over any
individual painting,
you can pick it out of
the space of color trends
over the course of Rothko's career.
I'm gonna turn the
transparency feature off now,
and then size these images
back down to a normal set.
Now we can see the individual paintings,
no longer overlapping.
Let's look at another axis.
We can cycle through all
of these various axes
and perhaps arrive at one
that has a particular shape.
Or has an image, like this one right here.
We can see that if we size
that particular image up
that's standing out from the
graph and choose to look at it,
we can see that this
one particular painting
is quite unusual in Rothko's career
for one or another low
level statistical reasons.
Now this doesn't mean that we
have to do all of art history
or all of visual analytics based on
low level mathematical statistics.
But it does mean that the
graph becomes an occasion
to pick out an image and say,
"Oh, this breaks the pattern."
Or, "This is typical of the pattern.
"Why, what's so particular
about this image?"
The important thing about this software
is it allows me to
explore transparent views.
It allows me to look impressionistically
at the way the data explodes
in a more complexity
at one side of the screen,
and focuses off on the other.
Let's go back to a tile view of looking at
all of these images again.
- So, you got the idea.
So basically we wanted to build the tools
which would allow us to explore patterns,
or an image or video
collections of any size.
And we thought, well, why
don't we use computers
to extract various
characteristics from images,
brightness, color, number
of shapes, texture,
and then create another software tool
which would allow us to sort
this collection realtime
naming any of these characteristics.
I'll show you next some projects
we've done more recently
where we use these techniques.
But before that, I want to
also demo for you one where we
more recently merged companies,
whereas now a few companies
which actually offer
computation for the masses.
And the people, when I
think about computation,
digital image analysis or contribution
of this thing that means the same thing.
You say, "Lev, so what are you guys doing?
"You probably guys doing
facing techniques, or maybe
"you automatically recognize
the subject in photographs."
Well, in fact, we're not
really doing that that much.
Because of our computation has developed
since late 50's, in fact
where basic techniques
were developed in the 60's,
but it only became possible
to process millions of images very quickly
in the last five, 10 years.
If you have an image,
and you want to measure
average brightness value, you
can do it for every image.
You just look at every
pixel and divide the number
by number of pixels, so
when you take a photo,
and you guys all have a
sort of laptop cameras,
and the camera shows a histogram?
So the computer's basically
doing the same analysis,
it analyzes piece and
value of the image where it
shows you a graph and a
visualization of pixel values.
You can do this, you can use
these techniques with every image.
But if you want to do, you
want our computer to do more
what's called high level analysis,
try to recognize faces, maybe
age in the photo, gender,
that still works pretty
okay, but if you want to
ask the computer to maybe
recognize what type of scene,
what types of objects represented,
it's getting better, but not that amazing.
So I will show you also
one of our projects
we're already using, for
example computation analysis
from the same company,
it worked really well,
and I think in this year that
maybe we can try to experiment
of scene analysis, but
it's a very very new one.
If you want to find two
passenger cars in the photo,
maybe it works with 90% accuracy.
If you want to find red
flags, maybe it'll be 13%.
But the field is developing,
so perhaps when a few years,
it would work quite well.
And obviously things
like face recognition,
I build anywhere, I build
it in all the cameras
from iPhone or iPod or
the most expensive Nikon.
I mean things like Facebook
also recognize faces.
So face recognition works pretty well.
Then you can see how what kind of things
the computer can instruct.
By the way, if you want
to use this company,
it's just one of them.
I'm not paid by them, but
I'm showing you because
it's the same company we used to do
one more project on the selfie.
Most that I've given to you, I give it you
to do some free processing about paying,
if it's a small number of images.
And, oh, oh my god, it just changed.
I'm sorry, honestly, I looked
at this a few days ago,
and it had, I guess with a
company just starting out,
you could get a free account,
so you're able to process
5,000 images for free.
But now it's already $99, but
okay, but $99 is still a lot,
so you can basically get
that out for a month.
Otherwise 45 images in the dialogue.
Or for $1,000 you can
analyze one million images.
Which is probably all the photographs
in all the museums in the world.
If you want to find out what
really happened in photography
it will only cost you
$1,000, of course you also
have to have some good
ideas, to know what to do.
So how does it work?
This is what we have.
The computer looks at the
face, analyzes pixel values,
and eventually gives
you lots of information.
First of all, it says I'm
confident there's a face.
Value is one.
But what's interesting,
when the computer gives you
this answer, that's also
very important to understand
about contemporary
computer based knowledge,
all sorts of knowledge, but
the knowledge is probabilistic,
it's not certain, the computer
gives you percentages.
It says, well I think the lady's white,
I'm pretty sure, actually 97% sure.
The age I think is 25.
The smile, I'm pretty sure, it's one.
So if somebody says,
I'm sure it's somebody,
it gives a probabilistic value.
And the thing about computation,
if you want to use it in your projects,
it works very well if you
give it these perfect images.
Here's a perfect face
on a perfect background,
but there's nothing else.
But if I start giving it
more real life images,
like what you have on Instagram,
it may not work as well.
Here is just some collection
of random, several images
from our project, so we
can try it out and see
if it's going to work or
not, so we'll just see.
Okay, so that actually seems like it's,
for example it found those
faces, but it missed this girl.
Sorry.
What about both of these guys.
So, okay.
I think that's Slavoj Zizek, actually.
So it must think he's not recognizable.
Even the very simple thing,
which computer scientists
have been working on for
70 years now, 60 years,
face recognition, it
works, but not perfectly.
Which of course has huge
implications for privacy,
you know how America has
one million people on this
dangerous list, so if you go to airport,
maybe some of you,
particularly if you have
a foreign citizenship,
you've been pulled out,
cross referenced with this list.
Obviously in most cases, people
are pulled because of errors
but now you can see why,
now you can see the dangers
of our society switching to computers.
If you want to use
computer to do for example
understand the content of a scene.
Yeah, if you give it this
perfect image of a beach,
it says, I think that's
a beach, 46% accuracy.
But it says, I also think
maybe it's a mountain 7%.
How you can use this?
And again, if you start
giving it some random images,
we can just try, I mean I'm
not really sure what happens,
so let's just give it some
random Instagram image.
That's pretty good,
there is a food, 4.46%,
but it also thinks maybe
it's a restaurant 1%.
I'm saying this because there
are some people who've been
looking, seeing my work,
and look at our lab and
saying Lev, aren't you a formalist?
Why you always organizing images
by these formal features like
color, bright, and so on?
I say, well, I would love
to do content but it's sort of hard.
Even the state of art software
you can see makes mistakes.
Now what I gave you, I
condensed the whole course
into 10 minutes, but you see
maybe really really basics
of what a computer can do,
how a computer analyzes images
and how we can use visualization
trying to use results
of this analysis, I will
show you some projects.
First we spent a few years
applying these techniques to
rather small image collections.
Actually very famous type of
images because we wanted to
make sure it was techniques
that were going to work.
Here is just one example.
According to the best
estimates, impressionists,
which is artists which participate in the
impressionist exhibitions
with various, 1970's, 1980's,
between 13 people, 15 people and we got
about 13,000 pastels in paintings.
Obviously there's not a
single book where you can see.
When you guys think about impressionism,
you think of wonderful flowers,
or maybe you think of Monet water lilies.
But you think of something.
Maybe like this, really
light, happy, Instagram-y.
Kind of Instagram, kind of vulgar.
I mean, it's a very vulgar art,
that's why it's so popular.
(humming)
Anyway.
(laughing)
So what we're done is, my students,
we spend a couple of
classes, and we collected,
we tried to collect all
images of impressionist
paintings we could find online,
so we couldn't find 13,000,
but we did find about 6,000, which is 50%.
And then we use computation
to automatically measure
which is to convert various
properties of images
into numbers, so in this case
it was maybe 200 numbers,
which were computing stuff from image.
So the computer looks
at for example values of
red, green, and blue,
the computer looks at
line adaptation, the
computer looks at pictures.
And when you put it all in the algorithm,
and we're using, if you're interested,
these decals we're using is from 1901.
It's from the regular sites from 1901,
which just shows you how
much it's progressed or not.
It's called principal component analysis.
It's a work horse of
contemporary data science.
By using this, it becomes
a collection of objects,
in this case you have
collection of images,
and we want to organize
these images in such a way
that the images which are
similar are next to each other,
and the images which are
dissimilar are far away from each
other, and we want to do this
as perfectly as possible.
In reality, there is no perfect
solution to this problem
because images can be assembled
in so many different ways,
so it depends on parameters you use.
But this is one of many,
many possible outputs
of such algorithm, and
you can see it's making
a bunch of mistakes, so here's
a blue image next to brown
images, but if you zoom out,
it actually works pretty well.
And what you realize is that in fact
what people think of as impressionism,
which is pretty maybe
slightly more green images
is maybe about 30 to 40% of
what impressionists produced.
This is what we're starting to
find out when we apply these
techniques to various canonical
sets of artistic images.
If you think about a particular
artist like Van Gogh,
or particular say artistic
movement like impressionism,
it's about something, it's about X,
it turns out that in fact
it's all more part of
people produced, but these
images are disregarded,
they're not written about by historians,
or not in museums mostly
in all these exhibitions,
we can delete them from art history.
Art history, this is the whole thing,
and you can see surprisingly,
impressionists have produced
lots and lots of these very dark images
which in fact are very
very similar to what
the other people in the
19th century were doing.
This is more typical art of 19th century,
and this is art like impressionism,
and by organizing images by
these other similarities,
you get a good sense of proportion between
these images which are more famous,
and these images which are not famous.
That's the real picture of impressionism.
Yeah, sure, sure.
Lots of them are portraits.
It's probably images that were painted
in the early part of their career,
before they discovered Rembrandt.
Yeah, maybe also Manet.
But just look how many of them are here.
This is another example
of what impressionists
have produced, and they
produced a lot of it.
That's one example.
So now you can say,
well, that's kind of fun,
but can we animate it to see how
these things develop over time?
We created this animation.
This is the same thing, but in this case
we're just using one artist.
We digitized all these imaged
from a catalog present there.
Which were to print between 1905 and 1917,
just before he developed
his signature style.
And we said, how can we use
these computation techniques
to imagine an artistic development?
An artist is developing,
he's searching for things.
Eventually he's going
to arrive at his unique
style, his unique visual language.
But what is the space of his search?
Is it something similar
to biological evolution?
You know how biological evolution,
70% of Americans believe
in both this evolution,
and then 70& of Americans
believe also in god,
so I'm not quite sure
how it works together.
Our life is full of contradictions.
If you look at the other
side of the half, it's 35.
What we've done is we've
done the same thing.
We used the same technique,
just the most basic
technique of data science,
principal component analysis.
We structured about 50 different features,
which is numerical
characteristics of images,
things like texture,
number of shapes, colors.
And then we organized on these images
to get automatically about these images,
all the images he painted
in 1904, 1905, and 1917,
12 years, about 144 paintings.
And then we simply animated this.
What you can see is that
going back from his final few,
the images are going to
pop up in the same year
as they painted, and
then the year is going to
appear out of left corner,
and what you're going to see
is a possible visualization,
suggested view
of how artist is
developing, in looking for
his or her unique visual language
in the space of infinite possibilities
because every possible
image you can paint,
photograph, imaginable would
be somewhere in the square.
So let's see how it looks.
Again, I want to point
out this is not science,
this is modern art, this is suggestive.
My idea is not to come up
with a new interpretation
of what happened to
Mondrian, but the idea is
to latch our ideas, and give
us more ways to think about
both present photographic production,
and also cultural history.
I'm going to artist, you get the shape,
you see the year, it appears
in the upper left corner.
Very interesting.
Let's do a close analysis, a
close reading of this process.
In the beginning, Mondrian,
I think he's stuck.
You guys, if you want to become artists.
You going to some high school.
You have a slightly
decapitated art teacher
says you have talent, now you end up,
now your parents are paying for art,
I don't mean it's here,
I mean some art school X.
(laughing)
Art school X.
And now you're like, okay,
why do all my photographs
look like rubbish print, or like analog?
Well, Mondrian, he was
not a genius either.
He was a bad school boy.
He made those paintings, and the thing is,
if you look at in a museum,
you also are going to find
what's very similar,
but I think visualizing
in the most abstract way would in fact
imagine visual difference of the images
is translated into distant, I
think it makes more precise.
You can see that the
images are very similar.
You can sum it as landscape.
But the fact is, he
really became abstract.
But he didn't know that.
He was a boy genius,
but he didn't know that.
And the color's very similar.
So here now, in the early 20th century
when artists will take
20, 30 years to mature.
Now of course presentation is different.
You made one new photo,
next day you have a show
on green points, next day for an out,
and you end up at the hospital.
But take some time.
Probably, if you do it,
you'll probably never
get beyond this, so don't
you want to become Mondrian?
Anyway, so here's the
start, 1905, 1906, 1907,
oh my god, the guy's in his
40's already, what's happening?
When is he going to break through?
You can see he's become
a little more interesting
because space becomes a little bit bigger.
But it's still more in one place.
And finally, oh my god,
1907, oh my god, boom.
So here's this breakthrough painting.
But you know what happens?
You know how life, it
can change overnight.
So yesterday you're a successful lawyer,
and today, there is
undergraduate or art school ads,
they'll say to become a photographer,
but you're still secretly
taking photographs
of locals or something, or you
have been in a relationship
that's already giving you guys hook ups,
and you both have got the money.
I think artistic development is the same.
I find that it's very difficult
for me to wake up tomorrow
and start writing about
something very different,
or start making photographs
in a very different style.
The moment he makes this new painting,
he also sometimes going back.
For awhile, he was doing
what he was doing earlier.
And then the new Mondrian.
He's going back and forth.
But as things develop, he's more brave,
he's exploring a larger and
larger space of possibilities.
And eventually, we didn't finish,
we didn't get to his
most canonical images,
but this is where we got to 1917.
Of course, this is not precise,
in fact, this is suggestive.
You can also visualize
the same information
in different ways, but
I think it's interesting
because what you find out
is that as he develops,
his new images are
further and further away
from where he started.
So the difference in visual language
and we use the visual
parameters to show this.
By the time we finish, he's somewhere
here and here and here, very
far from where he started.
But I think it's also
interesting to see that
as he develops, he explores a
larger set of possibilities.
Let's say by somewhere,
1912, he makes images
which are here, all the way
here, and also all the way here,
very very different,
so he's not developing
along a single line to go
from point a to point b,
but in fact, it's covering a
larger set of possibilities.
And then a few years later,
he's arrived at the canonical
grand style which for better
or worse would make him famous.
I think it's very suggestive,
very interesting way
to look at the artistic
or stylistic development.
And of course you can apply
it not only to history
photography and digital photography,
you can also apply it
to individual artworks.
Okay, so now that I've
shown you some examples
of what you can do, let
me show you our projects
where in fact you can
apply these techniques,
or even simpler techniques to
try to understand patterns in
that vernacular photography,
the social media, visual social
media narrative, specifically
we focused on Instagram.
I'll show you five projects
very briefly which we did
in the course of a year and a half.
Now I'm trying to recover,
to write a book about it
because it was very fast.
For our first project, we
said we're just going to take
a first look, we don't
know much about Instagram,
but we know it's not
just selfies and kitties,
it's probably a whole visual universe
which is probably has reached
the millions, maybe more.
I think our wall was both to
understand larger parameters
of what these photographs looked like,
but also try to combat a popular opinion
that social media is just drivel,
Instagram is just for nobodies.
But you guys know of course it's not true.
But people on the street don't know that.
We downloaded, it was me
and one graduate student,
so the whole project was done on laptops.
We downloaded 2.3 million Instagram photos
from 13 global cities,
and then we set out to
visualize these photos in different ways.
Most simple way, in this
case we're not using
any kind of complexity in the designs.
We simply took all the
photos, which we shared
in the center of a particular
city or a data set,
and we simply visualized them.
In an obvious way, we're not
going to invent this way.
This was to organize left
to right, left to right,
top to bottom, or, sorry,
left to right, goes like this,
in order of their upload.
This is 53,000 photos
from Tokyo which were
taken over a few consecutive days.
But you can also zoom in,
so we built this interactive
website, you can zoom in, move
around, and see the details.
And what's interesting
is that you get this
sense of what I would
call a collective montage,
a collective city film.
You have contributions of
probably 20-25,000 users.
Anybody that's taking
images were uploaded,
so people all were in agreement,
let's all go photograph
ramen, or let's all go
photograph this, it's basically
spontaneous expression of our free will.
But of course we're not so free.
We follow particular patterns.
We get up in the morning, we have coffee.
I check out Google analytics.
That's what I do.
Maybe you're checking
Facebook analytics or whatever
when you go to school.
Et cetera, et cetera, so
what I find interesting here
is that every day that we like,
I mean maybe it's a bit hard to see here,
let me actually download this image.
All visualizations we've
done as a principal
you can download them at
the highest resolution,
50,000 pixels, and it
prints, it won't charge you.
It's all opensource software.
I just downloaded this image
so we can look at this at larger scale.
As you can see that it's got
nice patterns, day, night.
But it's not perfect because people also
share lots of light
photography during the night.
But what's interesting,
every day and every night
is a bit different length.
This length of the day
night is proportional
to how many photographs people
uploaded in a particular day,
so one day is longer, one day is shorter
so you don't get a perfect bar chart.
In fact, you get this
subjective, objective
social city unconscious
visualized spontaneously
out of those photographs.
That's one thing we've done.
And then the second thing we've done is we
made these 50,000 samples
in number of cities
and organized from, again,
using this very basic
visual characters which the
computer can get from images,
it's not hard to guess what they are.
In this case, for the brightness.
It controls how far the
image is from the center.
So is it light or dark?
And then average hue.
Yeah, this is actually
average hue, average color
which is you can add
all the colors together,
it kind of works, controls
the handle of the image.
So from zero to 360, and here you can see
a very broad strokes of which
part of the color spectrum
has been filled in, and what parts of the
color spectrum is missing,
and I think there's one thing
which is missing very very very clearly
because we see images
every time we see this.
So what is one color you don't see here?
Green.
So not much green here,
not much green here.
This is Tokyo.
Yeah, a little bit
different, actually more red.
In fact, more red, maybe more
night activity, you know?
In this new one, not much green.
Here comes the interesting question.
Here are the same visualizations.
And I show you patterns
which are on the one hand
very similar, on the other
hand, it's a bit different.
There's some difference,
and here's is a Bangkok,
so particular colors are missing.
This is Bangkok.
These kinds of images are missing.
You don't have as much red,
there's a little bit of green.
And here you have something else.
Of course we can use statistics
and statistic analysis
when looking at these tests.
If you want to scientifically
make a statement
in that these image
collections are in fact
similar or different
according to the colors.
But I'm not sure how
statistical dissection
will be meaningful to anybody
in the room, including me
because we also have common sense.
So here is a question.
All that those visualizations tell us,
the fact that the combination
of modern statistics
which are becoming more
similar on Instagram which is
the same app, the same
interface given to everybody.
You have these filters which
appear in the same order,
everybody takes the same
pictures, which is these pictures.
Plus maybe some visual inferences.
So all those factors which
combine localization,
organization, software interfaces.
Do we lead to a certain monoculture?
Where people are taking the same images?
Or are those differences significant?
Are those differences
significant enough to say
in fact, people are actually taking
different photographs in different cities?
Yes, you could do statistical
analysis but what do we care?
And actually I think there
is not a clear answer.
I feel the answer, it depends
on our interpretation.
The point of this visualization,
not to give you a clear answer,
but in fact to give you
some more detailed ideas,
but also to open the conversation.
This was the first thing we've done,
and of course it's a way bigger approach.
We didn't really
differentiate with the images
in terms of subject
matter or different users.
We only can look at colors.
So then in the next project, Selfie City,
we said now we're going
to focus on a particular
type of images, so in this case,
we simply are going to now try
to compare apples and apples
as opposed to just compare everything.
I had a larger team this time.
It's about eight people,
including some new
visualization designers,
data scientists, art history,
PhD candidates from CUNY
graduate center where I teach.
And the graduate students
are a combination of
data scientists, art historians,
and then people like me who
don't know anything besides
just in between all the fields.
We decided to do a project
on what my team thought
was an interesting idea like selfies.
We decided to do it in August of 2013.
And I've been trying really very hard to
convince my team not to do it.
Guys, this is over now.
None of my academic friends
are going to respect me.
They said, Lev,
we know what we're doing,
just sign the check.
So they convinced me to do it,
and of course we got
thousands of newspaper
articles about it, people,
I spent months and months
answering interviewers questions.
Now of course I understand what
all of them, popular medias.
You can understand, a person asks us,
how are you guys collecting the data?
Did you actually check your results?
At the time I said, because
I had thought we had tried.
It was very interesting, but public media.
Here we try to look at
this, try to compare
selfie photographs from
six different cities.
And again to maybe talk
about the same question
which is are we going
to find more similarity?
Are people taking selfies in the same way
regardless of let's say,
gender, ethnicity, age?
Or are we going to find
enough cultural difference?
To what extent Instagram,
which I think leads to
perhaps a certain uniformity.
For one thing, everybody
takes the same pictures,
and you're using a mobile
phone, which also limits too
the photographic techniques,
are we going to find
uniformity, or will we find
that despite this uniformity,
maybe imposed by the software
and hardware interface,
there's still is found
a cultural difference.
We have done a few
different visualizations
suggest, for example, we
can show you a few images.
Here's a selection of
images from different cities
which are sorted very very in
one way, by face recognition.
And we can also show you how
to actually look at images,
so you can crop them, and
then you can also rotate them.
You can go from normal
view, computer view.
Just of course it might look
like we have bad things like
NSA and in effect, this
technique is also used
for lots of evil purposes.
And in fact, facial
recognition, as I've shown you,
are dangerous because it
doesn't work very well.
So then what we've done is
we've created these graphs
which look like normal histographs.
Where we compare the cities by
a couple of characteristics.
In this case, it's a smile
distribution, smile to smile,
by gender, by city, but instead
of making typical graphs,
we can make these graphs
once again from images.
The idea was not to reduce
individual differences
into summary, into aggregate,
which is how statistics
and graphs are typically used.
But the idea is to create
some kind of displace
graphical representations
which on the one hand
allow you to see patterns
but still remind you
that these patterns
emerge out of individuals.
You can move around, you
can see all the individual
histogram images, and
then you can zoom out
and see the patterns, and here, a little
findings, reports that
you have to click on.
You see in Bangkok that
people smile much more.
We saw it in Sao Paulo, you can also see
there are more female than
male selfies because female
is the top area and males
are the bottom area.
In Moscow, we don't really smile.
I mean, now we smile even
less after what happened,
but we really don't smile in 2013.
When we wanted to be able,
we said well, if we wanna
leave a project with a
level of visualizations,
we said this is abstract,
this is maybe more interesting
to visualization designers
or to visual artists, but journalists
are not going to write about it.
We wanted to create a
larger window so that normal
journalists who are not
interested in visual culture
can write about this project
so more people can visit it
and then maybe more people
can look at these graphs.
What we have done is we have done this
very conventional purpose,
pure internet research type of
findings, and again the purpose
uses a very familiar graph.
So we say only 4% of selfies
are more female than male.
And of course that's what
everybody wrote about
because this is what
statistics people expect.
But for us it was a device
to have a bigger window
to catch more visitors
and we hope this ability
can go in and actually look
at what the project was really
about, which is interactive
interface with the build.
Where, again, we experiment with combining
representation of patterns,
these kind of graphs,
can affect representation of media
which designs in this pattern.
Here's a whole collection, we collected,
it took months and months
because we wanted to collect
data under the same condition,
so eventually we got
3,200 selfies, so 340
selfies from every city.
You can browse them.
And then here you can
select these filters,
and you can filter them by
different characteristics.
And of course you can now
use this same interface
in each collection,
for example potentially
you can put all the photos
from MOMA into the same field.
For example selecting all
the female selfies verus male
selfies, and now of course
we've got a precise number.
You can also say, let me
select all the for example
selfies from New York, and
now of these New York selfies,
we're going to select people
who are older, after 30.
Well, I mean, meaning Instagram old.
Not actually that bad.
Here's all the old people, not that many.
What's interesting is
that, this is a histogram
that shows overall distribution
of ages in these photos.
How do we get ages?
We put this into the
algorithm and asking people
to guess the age, and also we use the same
face recognition software
which was very similar.
Except software always
judges people to be younger
because we knew people wanted to sell it.
What's interesting is
that if I select Bangkok,
you see how I have a second
graph which overlaps?
So now I can consider that
the selfies from Bangkok
can be younger of the New York population,
whereas selfies from New
York tend to be older.
This is the calculation of
computer in a few months,
and all this other stuff is computer,
so we use the face recognition
software which I showed you,
at which point was still free,
and we extracted 20 different parameters.
For example you can see what
about all of those faces
which are tilting to the right.
They tend to be females,
which is interesting.
And most of them seem to be in Sao Paolo.
You can see how it works, and
then you can add these filters
so now I'm looking at old people here.
And then the really old people.
(laughing)
Again, obviously it's not about interface,
but this is the direction
in which I want to go
to be able to create interactive interface
which would allow us to explore.
For people who don't
have technical skills,
any collection of images
of individual size,
and be able to actually
see individual images
in the same time to see patterns.
I know I'm aware of time,
I should probably finish
10 minutes, five minutes?
Yeah, I will show.
Very briefly, I will show
you some more things.
This is the project I've done
also last year on myself mostly.
Here I wanted to look at even a more local
collection of images, more filtering.
There was a revolution in Ukraine, Kiev.
Ukraine is a country which
was part of the Soviet Union
for those people who
don't know, this is a lot.
There was a revolution about a year ago.
There were people occupied
in the main square.
There was lots of
confrontation with police.
And then eventually, after a few days,
the police ran away, and the
new government took place,
and then a few days later
Putin said no it's not good.
We're going to attack right here.
That was a project about this.
I loaded all the Instagram
photos from the square,
city, around the main place
where all the protests
were going on, and I said
what can we tell about
the city from these photographs?
As opposed to simply saying
how do people who are
organizing a political
revolt or revolution
are using social media?
How is everybody using social media
in the city during this event?
How does this event even register?
We've done lots of things, we realize,
we look at the combination
of images and texts.
And so I wondered about
a couple of things.
One is, again, this is the
visualizations I've done earlier
with all those other projects.
The revolution starts
here, it goes over here,
and by this time we have a new government.
It's interesting, you can't find it.
What's exceptional, it's
drawn, it appears in the scenes
of everyday Instagram photos.
Project is not really about revolution,
project is about interplay
between everything,
because maybe there's a
revolution happening right now
on Wall Street, military
coup, but we don't know.
We're still taking selfies.
And then the exceptional.
We use different techniques to do it,
and also commissioned a
bunch of different essays
from a bunch of people to write about it.
It becomes a platform where
analysis, visualization
and different critical
essays by different people,
both students and faculty
in different universities.
What I also want to show
you is another effort
that you can understand
what's inside these images
which is a computer class thing.
Here because again other types of images
in terms of low level
characteristics, color and so on,
we feed them to the computer,
and ask the computer
to automatically divide
all those 30,000 images
in which we shared in Kiev
in one week into groups of similar images.
The computer divided those
images into 50 groups.
Some are just complete
garbage, but some work well.
Here's a group of those
which are the images
which have the most citizens
with particular composition.
Here in fact the images go
so far, which are lots of
compositions of images, and here is one
variations of selfies pieces,
particular selfies pieces.
But the computer doesn't know selfie,
it just knows there's
something dark in the middle
and something light around
it, and it's interesting
when you can use this
methods where the computer
doesn't know anything about each content,
but simply knows the
images similarities by
colors or pixel values or texture,
you can find these
clusters of similar images.
And we realize that
something relatively small,
which is a collection of 30,000 images,
and the images which as shared
in an exceptional event.
But if you just look at
this organized images
one by one, the ones that are shared,
it looks like a complete crazy
super modernist montage plus.
Here is evolution, here's
event, here's somebody's eye,
here's somebody selfie,
there's the food, products.
It's possible to see what are patterns.
How prevalent are
selfies, how prevalent are
images of revolts, but by
using computer analysis,
you can fin islands of similarity.
What it also means is that
if we deal with dozens
of all these photographs,
we don't need computers,
you can just use your human eyes.
If you want to make sense of photographs
created by millions and millions of people
to find most interesting
photos on Instagram
or even to get a sense of what happened
in this part of history,
you can use computer methods
because once you go into
thousands of millions
of photographs, the eye can't
understand the patterns.
Finally, very briefly, I'll
show you our latest project,
which Adam already mentioned.
It's currently stored in
the New York Public Library
as an installation,
but we also try to make
most of the stuff we develop online.
What it is is that, here
we can see even more.
We go from comparing
different cities when looking
at the selfies to looking
at a big part of one city.
Here we can see the street.
What we have created is a presentation
of a symbol of New
York, which is Broadway.
We tried to collect this data
which would form this core.
We have here 22 million taxi
rides along Broadway 2013.
All the publicly shared
Instagram images along Broadway
from 2013 for six months,
so that's 160,000.
Twitter images, Foursquare,
eight million Foursquare check-ins.
But also census data from US census.
What we're doing now
is writing papers where
we're comparing veterans in social media,
Instagram and so on, to
create characteristics
of neighborhoods in terms of
income, ethnicity, and so on.
And be able to actually
put our installation,
even be able to turn our
installation into an app,
although it doesn't work as well because
you need to have a very fast computer,
very fast connection
to consider this mode.
But probably if you go to
Starbucks it will work better.
Anyway, and we kind of made a reference
to a product which is here.
How many faces?
Does this face belong to
the terrorist database?
In other cases you can
see where we are actually
not really showing you,
showing you average statistics
the thing is, that's the thing.
We haven't used this, only used some
objective parameters, for
example face identification.
They have no idea how this works.
But of course we let you know
because our company copied,
that's another danger
of the systems because
we can play both sides,
so it'll give you results.
Let's say it was the system which would
give you number of how
beautiful a person you are.
What's the difference of
whether you're hired or not.
And people hired based
on this application.
It's intention, only
tried to use the system
which we don't know how it works because
we have to write using
opensource code, work as well.
But normally we write our own code.
Everything we do, we know how
it works, but in this case
we used something where we
don't know how it works.
(applauding)
