>> 
JONES: Good afternoon everybody and thank
you very much for this invitation, for representing
someone from CERN to come in to explain to
you a little about what CERN has been doing
in sort of data processing for our accelerator
which recently started up the LHC. What I'll
try and explain to you a little bit today,
just a brief [INDISTINCT] what is CERN? What's
it we doing? Why are we there? What does IT
Department do? And we've got quite a few represents
here today amongst you. And sort of what are
needs in terms of computing for LHC, LHC's
accelerator? And then, what model do we put
in place for that over the last decade in
effect? And sort of the--you know, how does
that--what's the relevance of it, how it's
going to change in the future, what's the
importance for the computer centers, typically
CERN, and the other major computer centers
around the world who are participating this
computing model and how it will impact as
well we think can be relevant for other scientific
disciplines not just physics, okay? I should
say, of course, that it's me standing here
today but this work represents the work of
many people at CERN, many other collaborating
institutes that are involved in the LHC program
and also a whole load of people and organizations
around the world who have helped develop great
technology which is a basis of a lot of what
we're doing here. So, just a word on CERN,
first of all. [INDISTINCT] for more than 50
years now and put together in 1954; originally
12 member states in Europe. We've now grown
to 20 member states and they–-you see the
list here, the 20 member states, plus we have
a whole host of other candidate countries
to join CERN and observers. And at this moment
in time, CERN is moving from being the predominantly
European Center to evolve to become a global
center for physics. We have—-we have 2,300
staff in total. The IT Department represents
about 230 people, so roughly a tenth for that,
and there's a load of other students, personnel
are in there as well, associates, fellows
and so on. And we have a user community, lots
more of them, Goggle, I understand more than
10,000 physicists around the world. Basically,
these are the people that want to get a hold
of the data that's coming off our accelerators.
Okay. Our annual budget is a billion Swiss
franks. So, today, it's money that's basically
a billion dollars as well and that's paid
by the member states. We're not a profit-making
organization. We rely on money given by the
member states to complete our physics program.
And just to be clear, we're doing physics
research. We don't generate electricity, we
don't build bones. We don't do anything like
that. We're looking into physics and the basics
of physics, okay? So here's our use of population
as well. You see the member states? It's all
over which represent two thirds of the population
of people who actually get a hold of that
data and using it. But you see many countries
are there as well. It's only about 69 countries
all in all together are actually making use
of the results of CERN. Not may people know
it, but CERN is a lot of--one of the origins
around the open access movement as well; so,
one of the founding partners there. So, all
the scientific publications come out of CERN
are open as well. Inside the IT Department,
well, like many other places, we're structured
in that way as well, but the IT Department
provides the computing resources for all the
requirements inside the laboratory. And so,
if we go around--I mean, the first and most
important one is the security. So, apart from
the—-apart from the management, the administration,
they have security there. Security crossed
all IT staff, security [INDISTINCT] department
for around accelerators, for all access to
the sites and so on, and things like that.
I'm glad to say that we have Louis and Antonio.
Where are you guys? Stand up. Say hello. The
next area is user and documentation services.
Now, you might say, "What's--what does CERN
got to do with this?" But, in fact, of course,
we've been around for more than 50 years,
we have one of the largest physics libraries--online
libraries that exist. And so, we also distribute
a lot of software which is used for storing
repositories. So the CDS system, Ingenio,
which is picked up by lots of libraries around
the world and used to manage their own repositories.
This is open source software distributed on
a–-distributed by CERN, maintained by CERN.
And also what to do with this group, it also
has to do with the audio visual system. So
going around here, you see you have your video
conferencing rooms. We do the same situation
at CERN but, of course, we had to do it not
only within CERN but across with all our partner
organizations around the world as well. And
so we Carmen and Pedro from this group. If
you look on to the experiment support--now,
this is where we are getting close to the
real physics, all right? So the accelerators,
CERN owns the accelerators we put in place,
the accelerator program and then their individual
collaborations, groups of universities and
research institutes come together and they
build the individual experiments which are
accepted to run on those different accelerators.
And the experiment support group, their job
is to make sure that those experiments can
optimize and exploit the IT infrastructure
at CERN to actually take their data. And so,
we have Fernando and Edward from that group.
Please guys. Next is database services. As
you can imagine, we of—-we need databases
to manage and organize the science data, the
physics data, the meta data that's generated
or produced with that as well. But all the
other data that you can imagine in an organization
of 2,300 people, their pay slips, all their
documents, all that sort of stuff, all the
administration stuff, all that's done as well.
And we have Katrina and Giacomo from the database
group. Then we have platform and engineering
services. So, what this area is actually worrying
about is essentially the massive batch system
that we have at CERN. We do a lot of processing
on site. So here we have massive cluster of
Linux machines. We have LX batch, which everybody
knows and loves, which is the interactive
system for where we process a lot of the physics
data. And they do various services like that
as well. And we have—-where are we? Yes,
please, stand up.
>> [INDISTINCT]
>> JONES: That's my mistake. I put your name
in there, excuse me. I apologize to everybody.
And then the communication systems, well,
CERN is a big campus. We have many--hundreds
of buildings on the CERN site and several
hectares in size. And, of course, inside that
we are a wide local area network. We have
a fixed network, which is high speed network
to nearly all of the buildings in CERN. We
have the Wi-Fi network as well and we also
have the high speed inter connects to many
of the physic centers around world. And Virginie
and Jose are here from the communications
group. And then computer facilities, as I've
said, we have rather major computer centers
at CERN and computing facilities group are
essentially operating that computer center
for us. So, they worry about ensuring that
we have enough electricity inside that, that
the cooling works, they also worry about the
procurements of all the generations of equipment
that go for that center as well. And we'll
come back for them later I guess then. But
first we go on to—-let's go on to grid technology.
We have a distributed system there for our
posting of the LHC data, a grid system. And
CERN, we've contribute greatly to an open
software stack which is basically called the
middleware for the grid system. This has been
produced through a series of years, different
projects run by CERN, also in collaboration
with the European--money from the European
commission also with their colleagues in the
U.S. and Asia. And we have Lawrence and Ricardo
from the group here today. Another area, of
course, is the data storage services. So,
we have discs like you can imagine where we
store all the data coming off of the-—off
of the LHC experiments but we have a hierarchical
management system where we have tape robots
where we store a vast quantity of these data
as well. And we have Lucas and Ana from this
group. Okay. And then we finally come back
to the computer centers I mentioned and we
have Eric and Zhecka. So, with that, we sort
of thought to introduce so you know everybody
here and you can ask them any questions you
like. We have no proprietary secrets so you
can--you know, right. So, moving on to the
accelerator itself. So, of course, Google
promoted--thank you very much as well—-promoted
the start up of the LHC. And, of course, during
2010 was our first major data taking year.
Here you see an aerial shot of the Geneva
Region. You see superimposed on top of that
the ring for LHC, which is 27 kilometers in
circumference. Next to that, you see a smaller
ring, the SPS early generation of the acceleration
which is used a--basically as an injector
in towards the LHC. On the far right you could
see the airport and so on. And at the bottom,
below the concentric rings there is the major
part of the CERN site. As I said, it's a big
campus. And you understand why we have a rather
large local area network there, so. And, of
course, over this side as well, we have the
other part, and so in the CERN site where
a lot of the fixed targets experiments take
place as well. So, it's a very thing and it
spreads to the Swiss French boarder which
basically goes through here somewhere. [INDISTINCT]
and things like that. So, on that accelerator,
as I've said, it's not actually on the surface.
You can't see a painted line when you go around
Geneva. It's not really painted on the roads.
It's below, yes, up to a hundred meters at
some points and it was actually developed
or dug out in the '85, was it, when it finished
'85, '86 with the previous accelerator called
the LEC. We subscripted--we dismantled that
accelerator the end of its lifetime and then
built the LHC in the same time. What you see
there, we looked at it more in detail but
you can see some of the [INDISTINCT] inside
the--inside the accelerator. And there are
four major experiments, and two other smaller
ones, but four major experiments sitting on
the LCH today. And so, here we see them distributed
around the accelerator. So these are the points,
essentially, where the particle--the particles
collide, beams of particles come together
and collide and that's where we collect the
data from these four major experiments. They
are, in a sense, looking for many of the same
characteristics, physics characteristics but
don't underestimate, they're also in competition,
all right? They're all trying to vie to do
the best that they can. They're competing
amongst themselves to see who can come up
with these discoveries as quickly as possible.
Part of the--of the accelerator itself are
sort of cut away. Looking at the LHC accelerator,
we see some of the blue parts of the magnets
and, of course, the magnets go all the way
around at 27 kilometers. So it's something
like, sort of a thousand--1100 magnets. Each
one several meters long. If you just look
inside the magnet you could see there are
two beam pipes there. So of course, we have
two--two rings circulating the opposite directions
and then they are brought to collide inside
each of the four experiments this is shown
on the previous shot, okay. So they're accelerating
that way, 10,000 times of a second or more,
going all the way around. This whole mass
is cooled for super conductivity reasons in
order to have high enough magnets which we
need to concentrate, bend such high power
beams. We need super conductivity and the
ability to pass electricity without any resistance.
And to do that the whole mass of this, some
40,000 tons, I think or something like that
are cooled using liquid helium. There's 130
tons of liquid helium inside this machine.
That's why it takes us several months to cool
the machine down to work on it [INDISTINCT]
up again. So, when you go back to your flat
tonight, you've been out for dinner, you had
a few [INDISTINCT] you go back home, go into
[INDISTINCT] and you think, "I'll just have
a night cup before you go to bed." And you
go to the fridge and you open the fridge.
It takes you three months to take that beer
out, all right? So just remember that. Think
about us when you go home tonight and when
you close the door, another three months.
Yes. It takes that much time because of the
maximum material we have to pull down. And
it takes us by sectors. Which we [INDISTINCT].
The running site of this machine is very long.
Previously, we would run for several months
with the old accelerator, it run for several
months, closed down in the winter when we
do maintenance also when the electricity is
more expensive but now we have to run for
longer periods because we lose too much time
cooling the machine down, warming that as
well. Here is just a little old simulation
on what it looks like inside on one of those
four experiments sitting on the LHC accelerator.
So what is it basically, it's a big onion
ring and each layer of that onion ring is
looking at a different physics characteristic.
And each one uses different materials and
technology more to do that. But all of them
are producing data, all right. So what you
see here is the experiment, that says, it's
actions are something like that. And here
you see the beam line coming through it and
you see the two beam particles--beams of particles
colliding. And out of that, produce other
types of particles which are detected inside
that machine, inside that detector. So it's
pretty much like a very powerful digital camera.
All right. These are happening--these collisions
are happening at 40--40 megahertz, or 40,000
times per second. And imagine--electronics
in here means that we have 150 million centers
basically. [INDISTINCT] million pixel camera
that is taking 40,000 photographs per second
under our four experiments, right? So you
start to get an idea on how much data we're
talking about which we have to deal with in
real time as it comes off these machines.
Well, in fact, this dedicated hardware inside
each of those experiments which is very, very
close to, on the ground which we call the
Triggering [INDISTINCT] Acquisition System.
So, the special electronics takes it off the
experiment and into the first they will trigger
depending on which experiment you might have,
one, two or three levels--one--no, two or
three levels of triggering and we take it
off then. At this point, we have some dedicated
hardwares--specialized custom-built hardware.
Sitting in racks and down in the bit. And
it is going through--you can't see a whole
event. You can't see a whole collision. You
can only see a little part of it because of
the amount of time it has to make a decision
is very, very short. So you kind of see a
little bit of it and like a key hole, it has
to guess if that's the right [INDISTINCT]
you want to keep or whether or not you're
going to throw it away. It's a bit like spam,
you throw most of it away because it's all
spam. And you want to keep a little bit of
it, all right. So, you'll just make a selection
decision, yes or no, keep it and go shoot
to the higher levels of the--of the trigger
and each time we have a little bit more time,
still, of course sub-second, we have a little
bit more time and we see a bigger part of
the picture until we can get to the level
three trigger where it's the first time we
can run an algorithm which can look at all
the data from an individual collision, okay?
And with that, we then take it from there
and after that, it's the first time it goes
out to permanent storage. So we haven't stored
it permanent to disc until that point. Okay.
If you look at this graphic, you can see the
accelerator on the ground. You can see the
four experiments and this is where some of
the electronics is sitting, the level three
trigger is sitting at the top of the event
filter funnel and we have a five optic cables
taking the data up to here and then taking
the selected events back to the CERN computer
set. [INDISTINCT] centralizing , getting out
there from CERN. We can--we can record it
and we can also spread it around the world
for further processing and to access to those
many thousands of physicists who want to get
a hold of that data as quickly as possible.
So, to do that, I say, we use a distributed
processing model because the amounts of CPU
you will need for this is--essential we need
100,000 CPU's everyday to process the data
coming off the--of the four experiments. Together,
the--we're recording about 15 petabytes of
data per year of these four experiments. Recorded
what the data acquired or what you [INDISTINCT].
And so we have a distributed model that was
built or designed basically a decade ago where
we have [INDISTINCT] CERN center known as
TIER-0. So this is where data comes off intimate
computer center. The computer center is TIER-0.
And we have 11 specialized centers around
the world. And each one of these centers of--have
agreed to participate in the LHC program.
They dedicated sufficient resources--significant
resources to work on this. And what we have--so
you basically, there's one in each of the
major--major centers for physics around the
world. We have GridKA is in Karlsruhe or in
Germany; Rutherford in the UK, SARA/NIKHEF
in Amsterdam, in Netherlands; FirmeLab in
the U.S., TRIUMF in Canada, we have Bellagio
in Italy, Lyon in France, Barcelona in Spain,
Taipei in Taiwan, Brookhaven in the U.S.,
and the Northern [INDISTINCT] facility where
the northern countries come together and produced
a distributed center across the northern countries.
So these are what we call our Tier-1 centers
and we have high speed dedicated links--network
links to those 11 centers. And so the data
taken at CERN is also sent out those 11 major
Tier-1 centers. They have specific functions
that will also record--store another copy
at least one of the copy of the data. So now
we have the replication of the data in the
systems. So, if you lose one copy at CERN
or somewhere else we've got a copy available.
[INDISTINCT] Significant processing capacity
as well to actually turn to all that data
and analyze the physics. Okay. And they also
provide another function which is to service
many other regional centers which we call
Tier-2, Tier-3, where the end physicist is
[INDISTINCT] and sitting. So that's--there's
a guy seated there want to get a hold of the
data, want to analyze, want to do the work
[INDISTINCT] services to these guys. And so,
that's the whole model we have and there's
of course this--there's a whole operational
mode that goes with this service level agreements
about what's provided at each level and so
on and it's completely monitored all the--all
the while, 365 days a year. All together,
there's something like a 150 regular centers
that are working and using and processing
and contributing to the state [INDISTINCT].
Below that, of course, the networking is very,
very important to us. And then much work is
being put in place to work with the academic
networks around the world. So that the national
regional--the national research education
networks and many other Europeans countries
contribute to helping to make the system work.
So we have high speed links between CERN and
the major Tier-1 centers. We also have back
up links as well. And this proved very important
because in 2009, where we were doing basically
dry runs for the LHC, some of the cables were
actually cut but the system didn't stop. Because
of the alternative rooting, we could continue
to process the data at the same rate or more
or less the same rate. So, we continued--continued
operation. That's very important. We have
to be very reliable in that way because we
can't switch off the accelerator. We have
to keep going. [INDISTINCT] As you see, it's
been running basically for about 
a year now. 2007 was the year when we took
our first major data taping here. Here are
the results shown, sort of, event displays
from the four major experiments. They're all
very happy that they're gathering data at
far better rates than they expected with far
better quality than theirs which we've foreseen
at this state. It's a--it's a fantastic machine.
So it takes time to tune it, get used to it,
understand it, of course, and the detectors
themselves. So, they're all very pleased that
the [INDISTINCT] are doing it. And the data
they are visualizing here has been processed
and delivered to them by the grid system that
we talked about. So, if we look [INDISTINCT]
here. During 2010 we stored, as I said, 15
petabytes of data onto a permanent storage
from the LHC experiments. Okay. And we are
storing it up to the rate of 220 terabytes
per day during the heavy iron run which happened
in November, early December. Tier is zero
so in and out of the computer center. We have
two gigabits--gigabytes per second with peaks
up to 11.5 and the average is about 6 with
peaks up to 25. So, it's a very continuous
system. It's not just a short verse. It's
a continuous operation of the scheme as well,
okay. In terms of this processing that goes
on inside the--inside the grid system. As
I said, the physics model were fortunate in
a sense that we're talking about individual
events and they can processed independent
of other events. There's no relationship between
the events themselves. So you can separate
it out, split it out into individual events,
package those up into groups and send them
off to different computer centers to be processed.
And that's what happens on the grid system
at the moment and of course they need access
to the--to the data to do that. And so, at
the moment we're running roughly about one
million executions--jobs executions per day.
So, across those many sites, they're running
about one million jobs per day, which represents
more than 100,000 CPU days per day. So you
get an estimate of the magic compute capacity
that we're actually using. And at any one
time, this is the number of concurrent physicist
uses online. So we've got around about 2,000
people at any one time can--currently accessing
the data. And the data is distributed around
the world within a matter of hours using this
Tiered scheme that I've just explained. Now
you got a sort of a rough idea to what it
is but let me say how we actually got there,
right. We haven't just kind of bought this
system. Of course, you couldn't see it in
the catalog anywhere so we had to build the
damn thing. And the way it actually really
is it distributed computing infrastructure,
it provide the production analysis. So it's
not only analyzing the data but all the simulation
that goes on beforehand for the—-for the
experiments. And it's really three major things;
a collaboration. There's a collaboration of
all these resources being pledged by the different
institutes around the world. They have a long-term
commitment to the LHC. They signed a memorandum
of understanding and they're giving resources
to LHC. There's no money changing happens,
all right? It's just that people say, "We
want to work on LHC. We were--are willing
to contribute this semantic computing resources
and effort to make the thing work, okay. Service,
I'll try to explain to you a little bit the
operation of the system with all these different
services in terms of data and the CPU and
the networking. That offers a real service
which runs 365 days a year, 24 hours a day
and there are service level agreements which
are defined. We can't fine anybody because
they didn't give us any--with them--paid any
money but what we do is pretty much like any
many places now is we monitor continuously.
And every month, we know the performance of
each individual site and we compare it to
what they pledged and so on. And we compare
that and it's either name and shame or name
and glory, right, depending where they are
on list, yes. So that works quite well. And
then of course is the implementation. I said
it's basically a distributed grid system--a
collaborative grid system, all right, of contributing
resources. And that's the--that's the technology
we have today, all right? But as I said, it's
basically designed 10 years ago and it's been
put in place and tuned, made sure it could
operator at this rate. Of course, we want
to see now that thing's have moved on. How
can we evolve that technology so that we could
change implementation but still maintain the
service and the collaboration? We don't want
to lose that, all right. So, if you look at
that structure in terms of collaboration,
framework or service and distributive computing,
this is basically what it represents today.
At the bottom, you've got the iron, all right,
the hardware and so on down there in the networks.
The switching model, we have above it, the
various services over here and then the collaboration
itself and so on. So what we really want to
be able to do is somehow is you place this
distributed--evolve this distributed computing
services to something else moving towards
a Cloud style model as well. A lot of the
Cloud Computing everyone sees now has a lot
of similarities and origins in what's happening
with the grid computing. So we can see some
sort of evolution if we want to go in that
way but still maintain some of the advantages
that we've got with the grid system as we
move towards that sort of model. So how do
we have to evolve this data processing? Well,
basically, the thing is to make it sustainable.
If you think about the lifetime of the LHC,
it's been in preparation for 20 years before
it stored data, yes. And it will run for 15,
20 years as well. So, it's a hell of a long
time, right? And we need the commitment and
make it sustainable. There are going to be
many generations of programmers, operators,
electricians, physicist working on this machine
and there's going to be a massive amount of
data that's going to be accumulated over a
lifetime of machine as well. So we have to
be able to make it operate in a sort of--an
efficient manner that can be sustained. So,
there are many data issues around that. The
whole data management and access question,
how can we assure or guarantee access to this
data on a global scale over decades? How can
we make it reliable and fault tolerant systems
when essentially the hardware we're building
on is not reliable and fault tolerant? Many
of the centers that we work with including
CERN, we buy commodity systems, we buy low
cost standard, not gold-plated discs or CPUs,
right? And they have failure rates often higher
than the--than the--that the manufacturers
publish, right? And we have to make a reliable
system still depending on those unreliable
elements below. The data preservation as I've
said, we're making it for decades and ensuring
it's open access and adapt to the changing
technologies. When this stuff was designed,
it was single CPU service people have. Now,
you have--there are many, many core CPUs and
who knows, it might be the predominant technology,
it might be GPUs for the whole commodity world
in--within the next couple of years. Can our
computing model adapt to that? Can our algorithms
exploit GPU architectures as efficiently as
the--as the single processor technology it
was designed for? It was like this, okay.
So we have to adapt to those. The similar
thing with global file systems, all the physicist
want to be able to see all of the data from
their experiments as one big global Unix file
system. That's what they want to see. Not
that easy to provide in the June--mid 2000
years, on that sort of scale but now if you
look around, there are different services
which are making--impossible things have evolved.
And if you think about it logically, well,
things like Dropbox, isn't that a global file
system? Could we not have something like Dropbox
for LHC? Could that not work across all the
different centers? Just trying to think of
different ways of doing it, things like that;
it's virtualization. Many of the centers I've
talked about are already virtualizing their
computing resources. One of the groups in
IT department is specifically put in place,
Virtual machines with Linux flavors, with
Windows service and so on. But can we--can
we expand that? Can we get it so that the
physicist algorithms are wrapped up as virtual
images which we can then distribute around
these different centers and allow people to
change and reallocate their resources more
dynamically than they can do today? And of
course, the networking infrastructure; when
this system was stimulated and put together,
the networks were not be particularly reliable
and people assumed in the simulation that
they would fail. It turns out there's been
such progress in the networking layer that
it's the most reliable part of the whole system,
right? So that sort of turns on the head the
way you have to think about it, right? Do
we still need this Tier system? Is a Tier
model instead be more appropriate now as these
technologies change? Has the structure and
the networking capabilities have changed as
well? So all these things we have to try and
take into account. But, I mean, we have to
remember these people involved in this as
well, right? And this--[INDISTINCT] when we
talked to the physicist about this and they
say, "Don't touch it. It works, leave it.
It runs, leave it," right? But of course they
want incremental improvements. But if we're
going to keep the thing going and maintain
it for the decade and so, we've got to see
how we can do it. And one important point
is the data, the biggest error when things
are going to change are in the data management
and data storage areas. But of course, data
is essentially is the family crowned jewels
of the LHC, right? Think about it. Somebody
who started working at CERN in the '90s or
'80s when things like Atlas and the LHC we're
being designed, they've probably been working
15, 20 years at CERN. If they stay for the
lifetime of the experiment after the 20 years,
they spent the whole career on one experiment.
So imagine you spent your whole career on
the experiment. And at the end of it, somebody
says, "Okay. We're not going to look after
your data anywhere else. We're going to give
it to someone else in the Cloud," all right?
How are you going to feel? All right. You
know, you've been spending so much, so much
of your life trying to get this data, are
you prepared to give it to somebody else at
the moment to manage it, all right? So we
have to get over that issue. We've got to
assure them that it is possible that we can
guarantee access to the data and that it will
be around it--they will have access to it,
okay. There's a whole question of, if we're
going to this Cloud model; public versus private
clouds, you could argue that what we have
today is a private cloud. It's open to the
people inside the LHC community. They can
share the data between them. They can share
the resources and so on, but we're not paying
anybody any money, we're not paying the--an
outside company to run that. However, in the
future, you might want to do that so that
we could consider that a public--a public
cloud system. [INDISTINCT] situations, there
are a couple of things we have to think about
from a legal point of view. Who has access
to it? Depending where your data is residing,
it falls under a different national jurisdictions.
And different governments have different rules
about what they can do with that data. Do
they have the right to access it or not? That
we have to take into account. Does it match
with the collaborations that have been put
in place [INDISTINCT]? [INDISTINCT] terms
of the public clouds? What are the terms and
conditions of the service? When you read the
contract and sign it, then you read the fine
print, does it actually offer you all the
guarantees you need? What's the service level
quality? Did you actually going to get for
your money? So there's a couple of references
that--say can see the bottom [INDISTINCT]
good report, but those are things that are
going on into thinking that people at the
moment, trying to understand how can we profit
from these advantages and this technology
but still ensure that we can serve the community
in the best possible way. And, of course,
if any--if you do go in this direction, we
start using public cloud, nobody is going
use a single cloud, aren't they? [INDISTINCT]
a single disc supplier or single CPU supplier.
You're not going to take just one because
if they collapse, what happens to your data?
[INDISTINCT] you're going to have several
of them. If you have several of them, what
standards exist? Can you move your data between
one cloud system and the other? If one company
goes bust, can you take it out and move it
somewhere else? Can you play them off [INDISTINCT]
forces and things like that? All those questions
come into play and also for better identity.
For the moment, we have a federated identity
system across all of those centers. Using
X5 and I certificates, you're given a certificate
by your host organization and with that you
can go and access the resources on any one
of those machines. [INDISTINCT] case if we
have different cloud systems around, can you
use your same identity across all these things?
This is going to be very important for us.
Nobody wants to keep logging into these different
things, tight? So those are some of the things
we were working on and trying to understand
how we can address at the moment. Okay. If
we can come back now, [INDISTINCT] go from
the global view and come back and look just
at CERN, at the computer center. This is sort
of--I should point out that the CERN computer
center, while relatively big by research standards,
[INDISTINCT] 20% of the computing resources
which are consumed by [INDISTINCT]. [INDISTINCT]
of it is actually coming from the other centers.
The Tier 2 centers, which appeared on that
graft to Tier 2 and Tier 3 as the small [INDISTINCT]
at the end, they are actually providing the
majority of the computer research system.
Sixty percent on average comes from those
small Tier 2 and Tier 3 centers and CERN itself,
as I've said, only provides 20% of it. We
can't possibly put it all in one center originally
because of, let's say, funding issues. CERN
is funded by all these member states. The
member states would rather than give us extra
money to build a massive computer center at
CERN than rather keep the money and invest
it locally in their own countries, which is
easier for politicians to do than to give
it to some other countries, some other place,
right? Invest it locally and then use it as
well for other scientists. Not just for [INDISTINCT],
right? And, of course, we have to build that
into the model that we use as well, okay?
If we look at it now, these numbers-- about
three months old [INDISTINCT], so it's continuously
growing, right? But we have about 8,000 servers
and 13,000 processes in the computer centers.
There, with about 50,000 cores, the [INDISTINCT]
don't worry about that. That's a CERN specific
[INDISTINCT] physics specific measurement.
As about something--over 50,000 discs there
as well. And you see, you know, this continuous
generations--different generations of the
hardware. We don't buy all at once, of course,
like [INDISTINCT] center. We are buying different
generations. We are trying to get the best
bank for your--for your buck so to speak in
terms of CPU capacity and so on and networking
[INDISTINCT]. So, of course, there are many
generations of the hardware in the computer
center. On time, we have to manage those points
as well. Continuous breakdowns of the equipment,
discs and power supplies are most weakest
parts we have there in terms of [INDISTINCT].
And, of course, we have [INDISTINCT] tapes.
We have about 150, 160 tape drives with the
robots from IBM and Sun StorageTek. It was
about 45 petabytes of tape storage there at
the moment. And then you see the networking
capacity that we have inside the computer
center. This computer center is quite an old
building. It's not a nice and new like this
lovely building we're in today. It dates back
from 1972 and it's about 5 megawatts in power.
It has a power utility efficiency of about
1.7. So we're using almost as much, again,
in terms of cooling and for--and [INDISTINCT]
sort of CPU cycles to get out. If we--if few
want to improve those numbers, we need to
have a new set-up for the computer center,
and CERN itself at the moment is looking to
have a remote Tier O, finding in some other
country in Europe who's willing to provide
a new computer center for us where we can
put extra capacity on those work, okay? But
if you look at the computer centers itself
this is roughly sort of the numbers game of
what's happening at the moment. From the experiments,
we get about 700 megabytes per second. It
comes into the--into the Castor system where
we have the disc pools which basically the
front pash to the, to tapes system as well.
There, the clever Castor software decides
when it goes out to the tape service. Here,
we're talking about less than half a second
access time. With the tape service, we are
taking minutes before going in and getting
out of data but it isn't just archiving the
tape system. We are actually using really
an active--an active part of the hierarchical
management system. In the future, who knows?
We'll see how things evolve there as well,
all right? Then from the analysis, the algorithms
are being run on the sort of the--on the service
and 
so on, then access in the data here as well.
And, of course, it's read backward and forward
here and then, of course, pumping data out
to the Tier 1 centers. Those are the rates
as well. In terms of the--sort of data management
layers, well, from the [INDISTINCT] physicist
user, he's got sort of a popular [INDISTINCT]
they use. It's called ROOT, sort of object
[INDISTINCT] framework for analyzing the physics
data being plug in their algorithms, do cuts,
fix and so on, [INDISTINCT] graphs and so
on after that, access the file systems through
a network client down to the network server.
The namespace, it looks like a big file system
if you want to the discs, from the disc staging
area and schedule it down to the tape subsystems
and offline storage because that's basically
the whole model looks like inside the Tier
0. [INDISTINCT] specifically in the data management
area, is that likely to move? Well, yes. Can
we take an approach where we move more the
scale, performance, and reliability issues
to a software layer? We are counting very
much on the hardware at the moment. Can we
change that? Can we make it more service-oriented
software layer in that sense as well? Looking
[INDISTINCT] scalability, we need to store
at least 20 petabytes per year. The physicists
are running faster than we are. The accelerator
is getting better everyday. The energy, the
luminosity is going well. The detectors are
getting better. They're understanding better
their machines, the tuning are better, understanding
how they can work better. So that's increasing
it. Twenty petabytes per year is looking conservative
now for what we have. And so, of course, over
lifetime of LHC, we're talking exabyte scale.
[INDISTINCT] we need fine grained access control
to all the different files where we store
multiple events and so on. And also we need
multiple authentication systems. Not just
X5 and I certificates. We need different ways
of integrating different systems there as
well. Accounting and journaling, that' very
important. We have monitoring and logging
systems at the moment but we can't roll them
back and restart as we would like. Global
accessibility; basically, all data has to
be accessible to anybody in the world that
has an internet connection [INDISTINCT] the
bottom line of what we have to be able to
do. [INDISTINCT]. And then the manageability;
sort of how can we limit the interventions
that are needed in the computer centers. How
can we-–at the moment, there are engineers
going on-call 24 hours a day and making sure
that the hardware is performing correctly,
all right? Can we change that so that we limit
it just to office hours and only using technician
level people to do it by making better use
of the software layers and so on? What can
we do in terms of sort of the multiple variable
levels of service quality? At the moment,
there's one level of quality. That's it for
the LHC, right? Can we have multiple levels?
Can we vary them according to what the needs
are, what we can afford, things like these
as well. And as I said as well, we sort of
trying to do all these with low cost hardware,
right? Commodity low cost hardware where price
is, again, a new factor. And, of course, the
power efficiency question. Everybody has to
be efficient with energy now. What can we
do? Can we turn off discs and manage the power
for them? With many core CPUs, can we shut
down some of the—some of the cores, what
can we do in that sort of area as well. [INDISTINCT].
Everything we're doing here is not just specifically
for physics, right? As I said, it's a general
purpose grid system evolving into a cloud
system. The data management issues are very,
very important. And this European commission-organized
high level group of export--experts put out
a vision for a Scientific Data where basically
you don't see the infrastructure anymore.
You just see the data. And that is a valid
for many of the scientific disciplines. At
CERN, we have strong relationships with people
working in health systems, life sciences,
drug discovery is a big use of this [INDISTINCT]
information and grid systems as well. Similarly,
in the environmental sciences and the earth
sciences processing the satellite data, [INDISTINCT]
as well. A lot of that is done with [INDISTINCT]
here at CERN as well. And so all of these
sciences are looking at these questions as
well. I'm that there might be some common
elements in what we can do here. And so there's
a basic model that people revolve towards--you
know, this is very, very--any software system
you can break into three layers. We can [INDISTINCT]
that, right? But if you--if you look at [INDISTINCT]
service, these people are trying to provide
here, that was the sort of things which we
can map on to what we have today and what
we want and, of course, the whole question
of continuous trust across all those layers
and the data curation as well. Let's finish
there. So thank you so much for your attention.
>> One from Grant, who contacted me yesterday
from Seattle. He read your 2007 paper on data
corruption at the LHC. He's wondering if you
have any follow-up?
>> JONES: [INDISTINCT] what paper that was.
Data corruption, we have--anybody familiar
with that paper? Otherwise, I will stab at
it. Okay, we'll look--I'll guess at it. Why
are you doing that? So, corruption could happen
in many layers in the system. First of all,
from the experiment themselves in the--in
the electronics level, all right? And that's
why we run a lot of–-a lot of time we spent
with cosmic--cosmic rays, when the [INDISTINCT]
isn't running. They're actually testing the
electronics in the detectors to see if they
archly producing the data that is expected,
right like what it is. Seeing if different
parts of the texture are dead or alive, all
right? Because when you assemble this thing,
there are zillions of miles of cables in them
and I guess or sometimes you don't connect
them quickly, all right? Then it causes a
triggering data or acquisition system itself
which may have hardware risk, it may have
software failures, it may have problems in
the algorithm, it maybe refer in the good
data, all right, which is one of the issues
to the experiment. And one of the reasons,
for example, at the start, when they start
something, they run with rather lose triggering
algorithms because they want to make sure
that they're not losing anything important,
things like that, does the--does the distributive
system itself. Now, in terms of that, yes
we can--we could potentially lose data but
as I've explained, we have replication of
data. That's very important. We use a lot
of other oracle facilities there as well to
actually distribute and have replicas across
CERN and the TIER 1 systems. And in terms
of the jobs which are running and processing
that data, if any one of those jobs fails,
they are monitored. The experiments note about
it. And the job is either restart it automatically
by our resource broker or the experiment framework,
software framework could itself will capture
that and reschedule it themselves. So, those
systems can fault tolerant build into all
those systems but we can't guarantee a 100%.
>> In the diagram as you mentioned that people
getting access to data with the Tier tool
with what kind of Tier it was of the uses
of the other end of physicists. The physicist,
[INDISTINCT] time slice if they ask for specific
contractions of particles they ask for specific
experiments and do they get the data or do
they simply request their algorithm is run
against that data somewhere else they asynchronously
and then have the results sent back to them?
>> Okay. So, there's--there's different aspects
to this. One is first of all, the collaborations
have a body of uses--physicists, right? They
are all working, contributed to that experiment
and they have--all are accessed to that data,
all right? But that doesn't mean people from
experiment have access to another experiment.
Remember I said during competition. So, they
are working to do something together. So,
they have access control over who can see
the latest data coming after the experiments,
okay? In terms of requesting that that data,
typically what happen, when they're reprocessing
the data, they are generating summary information
of--from the raw data. So, they're reconstructing
the tracts, the different characteristics
of the--of the events, different particles
and so on and that data is also written to
storage. That's why the amount of data that
you get written to the storage is more than
the data that is initially recorded on the
disc and so on. So, when they are doing their
analysis to things like role--root and so
on, they are not actually looking at the raw
data initially. They are looking a--basically
on a statistical basis. They are looking at
the summary information doing fits and cuts
on that summary information. Yes, sometimes
they are looking over [INDISTINCT] and for
each run, there's specifically called calibration
data from each of the experiments and that
records basically lots of characteristics
about what did the detector look like when
we took that data, all right? Is--having the
channels were on, off with a different subsystems
working, what was the timing situation across
the whole detector and things like that? So,
they bring all those thesis together and if
the characteristics are correct along with
the beam characteristics are correct to the
type of phenomenon they are trying to look
for, they will bring--narrow down the amount
of data they want to and do their analysis
based on that preference.
>> So, the question is, so basically, what
we're talking about is layers of process information.
So, mentioned there's this first layer responsible
for removing like extra information which
I think is interesting, okay? So, for example,
like if 10 years, I don't know, a physics--like
new model or something, going to look for
new like--I don't know, particles of new properties.
So, do we need like change this hardware layer
completely or we could reprogram it?
>> JONES: Well, there's two ways to look at
that, all right? There's triggering layers
all the way from trigger level one, two and–-one
and two. At that point, we haven't permanently
recorded the data, then it's lost forever.
Yes. It's gone. It's gone into--mass storage,
we have access to that data if at certain--under
certain conditions, you know, I took a calibration
data and so on for the-–for the experiments.
What could happen, you're quite right, is
it late to people to come back and say, "Okay,
I want to look at that data again because
I want to replace other algorithms to it."
They can do that but you can't change the
conditions under which that they do a recording.
It's done. That's past to it. You can--and
this is done dynamically and is part of the
online system, for each of the experiments
uses some of the CPU time of resources to
do this, it can monitor in real time what's
happening in the experiment and make some
adjustments to how the experiment is configured,
okay? But it's short term without doing anything.
Now, in the life time of the LHC, what's expected
is that, we are running basically at 7 TeV
now is a two beams of 3.5 TeV is designed
to go up to 70--up to 14 TeV. So, two beams
of 7 TeV but there are then plans as well
to upgrade the LHC to super LHC. So, this
really imply improvements in the technology
random some of the magnets, the radio frequency
and the cavities and so on, and the [INDISTINCT]
systems, to improve the quantity of data and
the quality of data we are getting out of
the--after the accelerator. Consequently,
there will be upgrades to some of the experiments,
for example, C&S [INDISTINCT], are looking
to upgrade in some of their systems as well.
At that point, we can start looking to other
characteristics which at the moment maybe
out of the visible range of what we can see
with LHC today.
>> Okay. Another question that I was sent
was around network availability stats for
long whole links, other ones that CERN operates
particularly. They were wondering where things
are filling in that isn't 100%.
>> JONES: Okay. I'm going to hand this over
to the networking guys. They probably know
better than me.
>> I guess, a lot of the [INDISTINCT] guys
were diggers and ship's anchors and earthquakes.
>> JONES: Yes. There's one particularly case
when there was the earthquake in Asia which
cut off some of our links. A lot of the links
not just for CERN but around Taiwan. But if--I
said, because we have the redundancy links,
we can---most cases, we carry on processing
as well.
>> So, you said these experiments are in competition
but you are sort of distributing the data
over all these different sites to encrypt
the data when you send it out or do you something
to turn off?
>> JONES: We have the access control mechanism
so this X5 and 9 based certificate system.
So, to access the data, you need those, right?
If you ask the physicists, they would tell
you, "We don't need to encrypt it, it isn't
encrypted because they don't understand the
dumb stuff anyway." So, it's true unless you
are really working on one of the experiments,
being able to interpret it--interpret that
data is very difficult. There aren't that
many manuals about the physics data itself.
>> How do you handle contemption for resources?
>> That's a very good question. As I've said
this, you have to remember that CERN is responsible
for building accelerators. One index RH of
facilities, all right? The experiments don't
belong to us. They are built by separate--separate
groups. The--first of all, can beat on paper
to be given the right to actually build the
experiment and have access to the accelerator.
And when it's build, they have a collaboration,
they have a--an agreement between themselves,
what they're going to do. Now many of these
centers that appear in the--in the grids are
serving multiple experiments. You know, researchers
just like to--like to hedge their bets as
well, right? So, they're going to work on
more than one experiment. And they say themselves
how they want to allocate their resources.
We do not impose it on anybody. What we do
from the CERN point of view is that we monitor
it in the sense of, we monitor what they pledge
to verify that what the centers are providing
are adequate for the experiments. So, if we
find that from one particular experiment,
there isn't enough resources we raise on alarm
bells, we can't force them to give it but
we raise an alarm bells and typically the
centers themselves will do something to try
and accommodate that. But the relative priorities
between them, is a decision for each of the
resource centers themselves, right? Typically,
they can do it in different ways. Most of
them got a batch system running. So, they
will give different cues to different priorities,
to different experiments, things like that.
