- Today on DevOps Defined, we're talking
continuous integration.
Let's go.
What's up everybody?
Ken M. Middleton here,
your DevOps Recruiter
with another episode of DevOps Defined.
Remember, if you get
anything in these videos,
don't forget to like and subscribe
so we give you a new
video each and every week.
So today we're talking about
continuous integration.
To do this, ladies and gentlemen,
today we have a very special guest.
We have none other than
Donovan Brown, yes,
Mr. Microsoft himself.
Donovan's going to share
a number of things today
in regards to continuous integration.
One, he's going to tell
us the benefits of CI
and why each and every company
should absolutely do it.
Two, he's going to share
the different types of CI
tools out there and
what you should consider
or be looking for in
regards to these tools
or as the future progresses, in regards to
what's going to be important.
Third, he's going to talk
about the one thing that
a lot of people do that
they think is smart
but absolutely ruins your
chance of actually implementing
CI into their environment.
A lot of stuff to learn,
so let's get into it.
- Hello ladies and gentlemen,
it is Ken M. Middleton here,
your DevOps Recruiter
here with another episode
of DevOps Defined.
Got a very special guest today.
I know a lot of times I
say I have a special guest,
but I really do have
a special guest today,
known as the Man in the Black Shirt,
known as maybe Mr. Microsoft,
Mr. DevOps himself,
as you can probably guess who it is
one thing you might not
know about this person,
of course he's an avid programmer,
but something that I found out
that I was very interested in
that he's ranked #11 in air hockey,
which I definitely have
some questions around that!
(laughter)
So that being said, you
can guess who it is,
welcome to the show today
Donovan Brown, from Microsoft.
Donovan, how are you doing today, sir?
- I'm doing pretty well.
I appreciate you having
me on the show, man.
I'm excited.
- Dude, so happy and
appreciative of you too.
Today we're going to talk about
something very interesting
and at the base of DevOps
that is very near and dear,
it seems, to Donovan's heart
is continuous integration.
With that being said, first,
before we go into that,
Donovan, if you will because
I'm always interested,
talk about your background in DevOps
and how you got involved in it,
and your, sounds like 5 1/2
year journey with Microsoft
with evangelizing it to the masses.
- Right, so I've been at
Microsoft for about 5 years,
but before I joined Microsoft
I was at a small consulting
firm called Notion Solutions for 7 years
and so I was doing DevOps long
before they called it DevOps.
I'm a certified scrum master
and as a member of that
consulting team, we would
fly all over the world
installing and implementing
a tool back then called
Team Foundation Server.
- Right.
- What Team Foundation Server
had in it was agile planning
tools, and continuous
integration and source control.
Being a certified scrum
master, you would stay on staff
with those customers after
we installed the software,
teach them how to use it,
and help them come from
the waterfall world into the agile world.
After you start producing an
increment of shippable code,
which is the goal of
agile, the next thing is
to ship that code into production.
You don't want to do that manually.
You want to automate as
much as you possibly can,
and that's a lot of what
DevOps is about, right?
DevOps is about people,
process, and products.
It's about all three, but a
lot of times you see a lot
of focus on the products
that we use and the
continuous integration and
the continuous delivery,
infrastructures, code, and
all those types of things.
When I came to Microsoft,
it was really starting to get a name.
People were calling it
DevOps and we were trying to
figure out how is Microsoft
going to position ourselves
so that we can go out there
and do some great things
for developers of any language,
targeting any platform.
We took Team Foundation Server,
which used to be something
you would install on
premise, and we basically
moved it to the cloud.
It's since been renamed to Azure DevOps,
but it is a suite of
services that allow a team,
again programming in any language,
to be able to automate the deployment,
automate the compilation
and testing of the products,
and automatically ship it
into production for you.
That's what I've been doing
now for the last 5 years,
5 1/2 years at Microsoft.
- Awesome, awesome.
So let's talk about this thing called CI,
continuous integration.
If you will, let's just start at the base.
What is continuous integration, Donovan?
What are the benefits of
it, as you see, in a DevOps
environment, watching companies do it?
- Great, so continuous
integration is the practice upon
every commit to the repository,
so a developer is writing code,
or a operations person is
writing infrastructure's code,
they're going to then
commit that to a repository.
That repository is where
you version your code.
It's where it's all safely kept.
Well, we can actually detect
when that check-in has happened
There's a trigger that
fires, and what CI does,
it says, oh, great! There's
a new change, let me go
download onto this virgin, clean machine.
Not a developer's machine,
a separate machine,
Let me download all the code,
and what I'm going to do
is I'm going to try to compile it.
I'm going to try to test
it, and then I'm going to
package it so it can be deployed.
The benefits of having
that is historically,
if you're working on a
team, and believe me,
it's unbelievable, in 2019
there's lots of people
that are going to watch
your show that do not have
continuous integration.
- Wow.
- For me, it's just
unbelievable that in 2019,
you're still on a team like that.
I'll talk about some things
that you can do to correct
that right away.
But what happens historically
is we would want to
a release.
They'd say, okay, Donovan
it's time to do a release,
and I would have to make
sure that source control
had everyone's changes in it, I personally
would do a get latest, make
sure that my machine was clean,
I would compile the code
and then I would package it
and test it and get ...
have a zip file or a binary,
and say, okay, who do I hand this to?
Who do I email this to so you can go off
and do the next manual step?
There's a lot of downsides to that.
First of all, my machine
might not be perfect.
My machine might not be
this perfect environment
to where I can make that
it's built even correctly.
Maybe I have old versions
of libraries on my machine
that are going to make it
behave differently in production
than what we expected it to be.
There's a lot of bad things
about having it built
by an individual.
Not to mention, I, Donovan
Brown, have to stop what
I am doing because I could
be in the middle of fixing
a critical bug, but they
want to do a release,
so now I have to find a way
to safely put my current work
aside, so that I can go
and then build this code
for everyone else so that
we can manually deploy it
out into production.
So as you can see, just
describing that sounds miserable.
- Yeah.
- When you put continuous
integration in place,
no one has to stop what they're
doing to produce a build.
Every person commits their
code to the repository.
Our build machine identifies
that there's a new change,
automatically downloads
it onto a perfect machine,
compiles it, tests it, and packages it.
One of the cool things about that,
if you go back to the scenario
where you do not have this,
I can say without any doubt in my mind
that if you live in a world
with no continuous integration,
you have come into work someday,
and you've done a get
latest and the code does not
compile in your machine anymore.
It's called breaking the build,
a lot of people make the
person who breaks the build
bring in doughnuts.
We usually have all these cool
raspberry pies (laughter).
That will tell you when
the build is broken so that
people know better, but
if you break the build,
you've now prevented
everyone else on your team
from being able to work today.
Because now we're all sitting
there, like we all did
a get latest.
Our code is broken, it doesn't
compile anymore because
Donovan broke the build.
Come on man, what's going on?
How do you keep doing this?
But the real question
is: how do we not know
that you've broken the build?
That's where CI comes in.
If CI is in place, when Donovan
checks in the broken code,
it's that machine that goes
off and tries to compile it,
and it instantly says I can't
compile this code anymore.
Something is broken, and it
notifies everyone on the team,
do not do a get latest right now,
because the build is actually broken.
So you keep the code that
you have, you're able
to keep working, while
I get a notification
saying that I just broke the build.
I go make whatever changes are necessary,
and I put those changes
back into our repository.
The build triggers again, and
hopefully the build succeeds
and sends a notification to
everyone saying the build
is healthy, everyone can get
the latest piece of code.
- It sounds like Donovan
it just saves you mounds
and mounds of time. Oh my God!
- It's a no brainer!
What's even crazier
than that is most of us
give you continuous integration for free!
Microsoft, CircleCI, AppVeyor,
there's just a whole list
of vendors who are like,
here, use this service,
it costs you absolutely
nothing to do, so why
are you not doing that?
That's the question that
I get a lot from people.
Donovan, I'm all on board,
I just saw you speak,
I love what you said but
they won't let me install
continuous integration at work.
I'm thinking, what do you mean
they won't let you install?
Well, they think it's going
to take too much time,
it's going to cost too much money.
I'm like, well your 1st
mistake, believe it or not,
was that you asked.
(laughter)
That is literally your 1st
mistake was that you asked
if you could do it, because
what's happening is,
we're all under a lot of pressure to ship.
There's always somebody
who's like, Donovan,
we needed it yesterday,
we needed it yesterday.
If I'm always feeling like
I'm behind the 8-ball,
or I'm always running behind,
I definitely don't think
I have the time to go off
and do something else, right?
I just need to keep on chugging
at this backlog of work
that I'm trying to do and
I don't have time to set up
continuous integration.
What you're not realizing
is, a 15 minute investment
is going to pay dividends
for the rest of your career.
It literally could take as
little as 15 minutes to set up.
You could get started with
our products or others
for absolutely 0 dollars and 0 cents,
and no one else on your team
has to know that you did it.
That's the kicker, right?
If I ... the reason I would
have to go ask for permission
is because I'm going to
inconvenience everyone,
or I need everyone to
stop what they're doing
while I go off and do
this particular work.
But if it costs you $0 to get started
and you don't impede
anyone else's progress,
why do you then need to
go ask for permission?
It doesn't make any sense to me.
So let's think about this,
let's say that we're on a team.
There's 7 of us on a team,
and I try to come to you
and convince you all that we
needed continuous integration.
Like, we don't have time for
it, we've survived this long
without it, let's just keep
doing it the way that we're
doing it because we've
got deadlines coming up.
What I would've done instead,
is I would have gone off
to one of these services,
I would've gone to dev.azure.com.
I would've created a free
account, and I would've
set up continuous integration
against our repository.
No one on my team would even
have known that I'd done it.
They would keep committing
code to the repository.
Azure DevOps is going to
light up and start building
the code for me and let me
know if the code is safe
or not for me to get.
So I just imagine.
Let's imagine it's Friday
and then all of a sudden,
Thursday night, somebody checked
in a build that was broken
and Friday morning, the team comes in.
Everyone does a get latest
like we always do, except me.
I don't do a get latest. Why?
Because I've got an email that
says do not touch that source
it's broken from the CI system.
So now everyone's at
the water cooler but me,
and they're all talking and
complaining about the person
who broke the build
while they're over there,
frantically trying to get the build fixed,
and eventually I hope
someone's going to say,
"Where's Donovan? Why
is he not over here?"
Like, no, Donovan's still
at his desk working.
They're going to say, "Well,
why are you still working?
Didn't you do a get latest?"
No, I didn't do a get
latest because I knew
the build was broken.
How did you know the build was broken?
Because I used a CI system.
What's a CI system?
You completely change
the entire conversation.
You're no longer trying to
convince them with your words
that this is a good idea.
You're letting the results
speak for themselves.
The fact that I'm still
working, and you're over there
getting water because
you can't work right now
is why CI is so important.
The next question they're going to ask is,
"Can I get in on that?"
Absolutely.
Give me your email.
I'll send a notification.
You'll know next time.
So stop asking to do your job.
Your job as an engineer is to
continuously deliver a value,
and implementing CI is going
to allow you to do that.
That is your job, so why are
you asking to go do your job?
To me, it's as ridiculous
as when you get to
a coding problem, I can
write it as a for-statement
or I can write it as a while-statement,
let me go ask my manager which
one of these I should choose.
I could write it as a
switch-statement or I could
write it as an if-statement.
Let me go ask my manager
if I could get permission
to write a switch-statement.
That would be ridiculous, right?
To me, it's that ridiculous
not to have CI in your system.
It is your job to do.
- Well Donovan, let me
ask, why are they asking?
Because to your point, I'm
under the impression that
it's part of what you do.
- Right.
- Why do you think some
engineers are asking?
And then I've gotta ask,
and you just kind of alluded
to this, is it the time
factor and the cost
is why the leadership is
thinking not to do it?
- It's two reasons.
I think the leadership
is more worried about
the time commitment, right?
Because the leadership is
thinking about milestones,
we have these deadlines we have to meet,
I don't have time for you to go off
and try to figure this stuff out.
What I tell everybody
is the amount of time
that it took you to
produce the PowerPoint,
or do the research to
go convince your peers,
you could've been done already.
That was a total waste of time
and then just be told, "No."
It's one thing, like,
if I go and tell you,
"Hey I want to set up
CI," and you say, "No."
and I do it anyway? That's bad.
If I just set up CI and
I didn't even ask for it,
and you reap the benefits,
it's all goodness.
Right? So asking actually
slows you down by asking
for permission to do something
that I think is your job.
But it's the time and the money.
Those are the two things that
I hear people talk about.
They think it's going to take
too long or cost too much.
Well I'm here to tell you,
we give it to you for free
and it takes about 15 minutes,
and 15 minutes is generous!
Let's say for example, you're brand new.
Never done this before,
it might take you longer
than 15 minutes, you still
don't have to ask permission.
I take 15 minutes out of my lunch,
I try to set up CI on
Monday, I get halfway done
and then I put it aside. Go
eat 45 minutes for lunch,
do my job. No one in that
building knows anything
is different than it was on Monday.
On Tuesday, give it another 15 minutes.
Maybe by Friday, I get it working,
and then from that day
forward, I reap the benefits.
Now if I was going to take 1/2 a day off
to go to try to figure out
CI, you probably have to
ask permission, because
whatever it is you're
supposed to be doing that
day probably isn't going
to get done because you're off
doing your little pet project
But just take 10-15 minutes
a day to go do a little
bit of research and figure this out
because no one's going to
fault you for that small
amount of time and you don't
have to go ask permission
to go spend 15 minutes of
your own lunch making yourself
better and making your team better.
- And when you do it, it sounds
like from your description,
that it's not going to
change, once you implement it
nothing's going to change
in your environment,
besides the people who take
advantage of the benefits of CI.
- Exactly! Your peers will never know!
Your peers will never know.
Your repository looks exactly the same,
they don't know that we're monitoring it,
they will have absolutely no
idea that you've done this
unless you tell them or
they see the benefits of it
themselves.
- Wow, that's a no-brainer. That's crazy.
- It is!
What's crazier to me is when
I go to a conference in 2019
and I say, "Raise your hand
if you're on a team that
doesn't have CI." And hands still go up!
Come on, man.
I've been writing software
for 20-some odd years.
I live right down the street
from the old compact computers,
it's HP now, that's where I started.
I remember how hard it
was to set up a CI system.
I remember how long it used
to take to set up a CI system.
We didn't have one when
I first joined the team.
We had one when I finally
left, but creating make files
and setting up a machine, and
getting the network stuff...
Our world has changed.
You click a couple buttons,
and CI is up and running.
It's just amazing.
- It's done. What are the main tools now?
I mean, I'm sure you're
a little bit biased,
but what (laughter) we want
to talk about the major tools
and the ones that you see work
the best in an environment
and let's talk about from
a cost-perspective versus
effectiveness-perspective,
let's talk about both of those.
What do you see as the main
tools out there that people
should take a look at?
- The #1 competitor that
we have, believe it or not,
is Jenkins, which is
a free product, right?
Jenkins is free.
You can go and download
Jenkins, install it, and start
building with it immediately.
Jenkins has little bit, I
think, more of a learning curve.
I install our competitors'
products, clearly,
so I can understand the
true cost of ownership
and if you have Jenkins running right now,
and it's been running
for years, you have now
forgotten how hard it was to set up,
because you're just reaping
the benefits, right?
You're like, Jenkins is awesome.
We're able to build, we can
make code, it's fantastic.
But like, do you remember times zero?
Do you remember when you
installed it for the first time?
I do because I re-install
it, like, monthly
to see like, has it gotten better?
What are our competitors doing?
So the initial set-up, getting
all the plug-ins in place,
learning a new tool,
takes a little bit of time
when it comes to Jenkins.
Azure DevOps, which is the
product we have at Microsoft,
one of the features is called Pipelines,
and Pipelines integrates with
GitHub right out of the box,
it integrates with Bitbucket,
it integrates with Subversion
so wherever your source code is today,
chances are we can go ahead and grab it.
It's absolutely free to get started.
For example, the 2 of us
can invite 3 of our friends,
go to dev.azure.com and
create an account for free
and all 5 of us can use
everything that we have for free,
including our CI system,
work item tracking,
our release management
system, our testing framework,
even our package management, all of it.
You get it all for free for the 5 of you,
and then the next person
you add is $6 a month.
Right? It's just ridiculously cheap.
Now, if you're using open source,
you're in GitHub and it's a private repo,
we give you 10 unlimited
Pipelines for free,
unlimited build minutes, and access
to Mac OS, Windows, and Linux.
Now that's really
important because depending
on the language you're
programming in or what
machines you're targeting
for your customers.
They might be on Macs,
they might be on Windows,
or they might be on Linux,
and you want to test your code
over all those sort of things
You don't want to just test it on Windows
and assume it works on Mac.
You want to test it on Mac,
you want to test it on Linux,
you want to test it on
Windows, and to do that
you're going to have the machines running
all those languages.
I don't know of any other
vendor that has all 3 platforms.
You'll find vendors that
have 2 and then you'll have
to go to another vendor for the 3rd one.
You might have a vendor
for 1 of each of them,
but now you're making 3
different build scripts, right?
Because if I use AppVeyor
for 1, and Travis for 1,
and Circles for 1, yeah they all use YAML,
but the YAML isn't the same,
the way you speak isn't the same and now
I'm maintaining 3 different
build scripts for 1 project.
- Wow.
- When you use Azure
Pipelines, since we give you
Mac, Windows, and Linux
machines, it's one definition,
but you just tell us where
do you want us to run it,
and we'll run it on all 3 of
them or any choice of those
that you want, so it
makes it a lot easier,
and again, if it's open
source, we're the most
generous open source offering
in the world right now.
Because we just bought
GitHub at Microsoft, right?
So we're like, come on,
welcome to the family.
We're going to obviously give
you the best offer we could
possibly give you from Microsoft.
If it's open source, it's free.
So Jenkins is hugely
popular, you were asking.
Azure Pipelines is becoming
very, very popular.
You're also going to hear
things like CircleCI,
Travis, and AppVeyor.
What you can do is just go
to any GitHub repository
and you're going to
see these little badges
on their Read Me file, and
those badges are the CI systems
that are compiling their
code, and you'll see
multiple of them there
and you'll get a good idea
of the most popular ones
just from the badges
you're going to see on GitHub.
- Gotcha, gotcha. So it sounds
like for, with Azure DevOps,
it's just easier to set-up and
it integrates, it sounds like
almost with everything immediately.
(laughter)
- Pretty much! We're trying
to change the perception
that we only work on
Windows and .net, right?
- Right.
- We can help you with
your mobile application,
we can help you with your Go lang,
we can help you with your Python,
we can help on your Mac, your
Linux, or your Windows machine
- Talk a little bit, because
I've got to be honest,
you're exactly right, I've
been under the impression
for a long time, whenever I
saw TFS and now Azure DevOps
that it was a .net shop.
But you're saying, any
language it doesn't matter,
you guys can integrate,
open source or not.
- I cannot remember the last
time I did a .net demo on stage
Can't remember, right? I go
up there and I either do Node
or I'll do Java, just to prove that point.
Just so that people can
see that I can actually do
any language and any platform.
There's a video of me once,
I did it once in Germany
and I think I did it
once in Australia, or no,
in New Zealand, where I was so confident
that I could do any
language, any platform,
that at the beginning of this talk,
I let people go to an app,
go to a website and vote,
tell me what language you
want me to program in,
tell me what platform
you want me to target,
and at the end of the demo,
I'm going to do whatever it is
that you want me to do, and I just go off
and start telling them about everything.
They're voting, and I turn
around and I literally had
a Mac on the desk, a Mac,
a PC, and a Linux machine.
They got to pick which machine I use,
they got to pick what
language I programmed in,
and they got to pick
where I deployed my code,
like I don't care what you pick.
This is what I'm here to
prove. I can do any of it.
And over and over again,
I just kept proving,
whatever you want me to program
at Microsoft we can help you
- That is awesome.
Now you said you were in New Zealand?
- I've been ... Yes, I was
in Auckland, New Zealand
for Ignite 2 years in a
row and I think it was the
2nd year that I went to
Ignite in New Zealand,
you can find that video where
it's the 1st time I do this.
I was so excited and just
buzzing to get on stage
and just, let me show you what I can do.
There's a cool little app
with a little South Park
picture of me, it's
hilarious, and the little guy
helps you vote and pick
what you want me to do
and at the end of it I'm
like, alright let's go.
- Let's go! Alright, well
I am going to put that
in the show notes so people
can go and click on that video
because I think they'll
be very interested in it.
That's awesome man.
Kind of to wrap it up, Donovan,
as we talk about the end,
what do you see as the future of CI?
As you look through the
next 2020 and beyond,
what are we looking at as
far as the features benefits,
where do you think it's going?
- Yeah, one of the benefits
has to be reaching all of the
different platforms at once.
We are gone from the days
of ... I work at Microsoft,
I go visit a lot of our customers,
very few of the customers I
visit only do one language.
Right? They all do multiple languages.
They're all into mobile
development, so you're going
to have to make sure
they have a platform that
allows you to target Android and iOS
and allows you to hit that Apple Watch.
For that, you're going to
have to have a Mac, right?
And if you want to do
some Windows development
or some pure .net, not
that core, you're going
to need a Windows machine for that.
If you're doing .net core,
you can now build your
docker images inside of your build system
but you're probably going to
need a Linux machine for that.
You can get away with it
on other platforms, but...
So you want to make sure that
you're picking a platform
that's going to allow you
to build on all of that,
and the future is more and
more of us are going to
start providing you with every
single platform that you need
We're just the 1st ones to do it.
Also, thinking about how
you're going to integrate
every part of your product
into your CI/CD pipeline.
A lot of times you'll go
to a customer and all they
have is the front end, their website,
but their database is still done manually,
their mobile app is still done manually,
it's like come on, your
entire solution needs
to be in that pipeline.
I think that we're going
to see a lot of people
enabling and making it much
easier for you to do CI
for your mobile apps,
CI for your databases,
CI for your docker images
and stuff like that.
I think we're ahead of
the game there because
we already allow you to do all that.
I am going to speak at
VSLive here in New Orleans,
and the demo that I do does
everything that I just said
with a single pipeline.
I build an Android and iOS application,
a database project, a .net webservice,
a .net core front end for docker.
I do it in 1 build because
I have all 3 platforms in
Azure DevOps, and I push 1
button and next think you know,
I'm deploying containers,
and databases, and mobile
applications, it's amazing.
So it's going to be awesome.
- How much time, oh my goodness,
how much time do you save by doing that?
And just, the lack of mistakes, right?
By automating everything?
- Exactly. Exactly because
you do that, a human being
does that and I only have
to mess up once, right?
And I could bring down a company.
There's a story about Knight Capital Group
where a poor guy who was
deploying to, I think,
8 servers, he did 7 of
the 8 servers correctly,
he screwed up on 1 server,
he lost $464 million in 45
minutes and bankrupt his company.
That's what happens when
you send a human being
to do a machine's job.
A machine would have done
that perfectly, 8/8 times.
This poor guy messed up 1
time, and lost half a billion
dollars and bankrupt his company.
- [Ken] Oh my goodness.
- So that's why CI is important.
Automate everything.
- There you have it, ladies and gentlemen!
I think that is a great
story to end it at.
Alright, Donovan, anything you
want to add as we wrap it up,
anything you want to tell
the audience out there,
any last words about CI?
- Yeah, implement it!
Stop asking for permission!
If you need help, if you
need help setting it up,
if you get stuck, you can
actually find me on Twitter.
I'm @DonovanBrown.
My entire team, we're called The League.
We're all on Twitter, we're
on Twitter all the time.
If you use a hashtag, #LoECDA,
if you use that hashtag in your tweet,
all 5 of us will come and read it,
and if we don't know the
answer, we'll @ someone
from Microsoft from the product group
that can answer your question for you.
So, if you don't have CI
or you want to switch CI,
come and reach out to us on Twitter
and we'll come help you.
- Say that hashtag one more time?
- #LoECDA. It stands for
the League of Extraordinary
Cloud DevOps Advocates.
- LoECDA, I like it.
It's probably going to get
a crazy influx of #LoECDAs.
- (laughter) Yeah we've
had it used, I think the
maximum was 900 times in a month.
I think was the highest it's every been.
- God, love it man.
But Donovan, thank you so much
for your time, my good man.
I appreciate it greatly!
- My pleasure.
- Alright man, take care.
Alright, there you have it.
Thanks Donovan for being a great guest
and hopefully everyone learned a lot about
continuous integration,
different types of it,
and it helped to give you a
little more knowledge there.
So as always, if you're
interested in watching
more videos of DevOps Defined,
take a look right here
as I'm including other
videos you can watch
from the different series,
and as always, if you're
looking for more career tips
when it comes to DevOps,
I've included them here.
So as always, thank you
everybody for listening.
Hopefully they helped a little bit
and happy continuous improvement.
