Continuous Delivery

I just finished the book, Continuous Delivery by Jez Humble and David Farley. It's a great book, but it is a bit wordy (512 pages), so here is the gist of it.

Continuous delivery builds on a simple foundation, continuous integration, automation, and version control. But in order to get it to work you must version control and automate everything. It's that simple! Version control and automate everything! Everything means different things to different people, but to Jez and David it means a lot.

You should version control:

• Source code and test code
• Assets, images, media files, base data, etc.
• Build scripts
• Deployment scripts
• Secret configuration settings, like keys and passwords
• Environments, development, production, test, staging, ...

The first three you are hopefully doing already, but the fourth is unusual and the fifth and sixth, I rarely see anywhere. Secret settings should be kept in version control too, just don't put them into the public repository. Here are some question to ask yourself:

• Can you set up a complete development machine with one command?
• Can you set up the deployment pipeline machine with one command?
• Can you set up a fresh test environment with one command?
• Can you set up a complete production environment with one command?
• Can you upgrade the application, including data migrations, with one command?
• Can you roll it back, in case it fails, with one command?

The Deployment Pipeline

Continuous Delivery uses a Continuous Itegration server to drive the Deployment Pipeline. In the simplest form of application, a deployment pipeline may have only one step. This step, runs all the tests and if all goes well it will deploy the application. But as soon as the application grows this will take too long, the single step needs to become a pipeline.

A deployment pipeline may have arbitrary many steps and they should be as automated as possible. A common setup of a deployment pipeline contains the following steps.

• The Commit Stage (automatic)
• The Integration Test Stage (automatic)
• The Acceptance Test Stage (automatic)
• Exploratory Testing Stage (manual)
• Perfomance Testing Stage (automatic)
• Deployment Stage (triggered manually)

The main reason for the deployment pipeline is to provide feedback as fast as possible. The commit stage typically only contains unit tests that run very fast. The steps after that only run if the step before succeeds.

The steps above may contain everything you need, too much, or in the wrong order. Feel free to remove and re-order as you please. If performance testing is cheaper than exploratory testing, in your case, switch them. There is no one-solution-fits-all when it comes to pipelines. Customize it, to fit your needs.

One important aspect of the deployment pipeline that I rarely see is that the final product should be created by the first commit stage and then resused by the other stages. Very often I see that the separate stages check out and build a new version from scratch, albeit a tagged version but still. The reuse of the same artifact through all the steps not only ensures that the same artifact has passed all the stages, it also speeds up the deployment pipeline, removing redundant work.

Another thing worth noting is that the manual steps described above should be as automatic as possible. If a tester want to test the latest build that has passed the acceptance test stage, he should be able to do it with one click. That one click may involve:

• Create a fresh production like environment.
• Install the selected artifact into the new envrionment.
• Apply all the relevant configuration.

If the tester decides that the build should fail or succeed, he should be able to do that with one click. And, the feedback with the result should be fed back into the deploynment cycle so that the developers know that it failed, and why, or the operation guys knows that there is a new version ready to deploy.

Conclusion

The book is a good read and I highly recommend it. In the process of describing continuous delivery, it provides tons of useful tips on keeping your code and data under control. Among other things it describes.

• Release strategies, such as canary relasing and blue-green deployment.
• Test strategies for acceptance tests, capacity test, etc.
• Why not to use long-lived feature branches and how to avoid it.
• Dependency management
• ...

No Deadlines

deadline, from New Oxford American Dictionary

1. the latest time or date by which something should be completed : the deadline for submissions is February 5th.

2. historical a line drawn around a prison beyond which prisoners were liable to be shot.

Deadlines in software development induce unnecessary stress, and transform otherwise enjoyable activities into chores.

We need to replace deadlines with livelines. A liveline is a deployed, working version of a product. It is after the fact, a milestone set when you are done.

How Deadlines Are Set

Deadlines are set in different ways in different projects.

In the classic Death March project, a deadline is set by someone outside the project with no clue whatsoever about how long it will take. Typically this kind of deadline is based on some external event, where the project will be demonstrated. The event may be a conference, or something similar.

In Scrum, the deadlines are set by the process itself. Most Scrum projects I have worked with use a two week deadline instead of the recommended 30 days. The team decides how many stories they can complete by that deadline, based on rough estimations done in the sprint planning.

Some projects set deadlines by a developer being asked (or asking himself) when the task will be done. The developer will say: "I'll have it done by 3 o'clock", a wild guess.

The problem with all these approaches is that the deadlines are, at best, based on guesses of the team doing the development. But guesses are guesses, not something you can set a deadline by.

And deadlines work both ways, a task is very rarely completed before it's deadline.

Kinds of Deadlines

Real Deadlines

If a task is not completed by a certain time someone will die. This kind of task is really common in novels and movies. A clock is ticking and if the task is not completed by the time the clock reaches zero, BOOM!

If have never seen this kind of deadline in software development, but they may exist.

Planning Deadlines

Some deadlines are set in order to simplify planning. In the large Death March projects, the plan is to show the project at a certain date. In integration projects, the deadlines are set to be able to synchronize with other teams working on the same product.

This kind of deadlines are actually understandable, but I am arguing that they actually slow down projects by introducing stress, which introduces bugs. The bugs introduce bugs in the dependent projects, and slows down many projects instead of just one.

Motivational Deadlines

Motivation deadlines are set to get people to work. The assumption is that if a deadline is not set people will procrastinate over different ideas, or just slack off since there is no date set for the release anyway.

This is, absolutely, the worst kind of deadline. If the team is not motivated enough to do a good job without a deadline, you need to get a new team, not set a deadline.

Why Deadlines Are Bad

Stress

Consider walking on a one foot wide plank. If it is on the ground it is easy. If it is two meters up it is difficult, but if you put it at twenty meters it is a whole different ballgame. That is what its like with deadlines, they put an unnecessary pressure on people.

Loss of Motivation

Work consist of whatever a body is obliged to do, and play consists of whatever one is not obliged to do. -- Mark Twain

In his book, Drive, Dan Pink differentiates between intrinsic motivation and extrinsic motivation.

Extrinsic motivation is motivation driven by carrots and sticks, rewards and punishment. It works well for simple, well-defined tasks.

Intrinsic motivation works well for creative, ill-defined tasks, like programming.

Intrinsic motivation is fragile, it disappears in the presence of carrots and sticks. A deadline is a stick, whether set by ourselves or someone else.

Livelines

A liveline is, quite simply, a task that is done. When a task is done, it is easy for others to use it, in their plans or in reality. If we want to build a stable house, we should build it on stable ground.

When using livelines the iteration length is the same length as a task. This is usually a lot shorter than the usual one to four weeks common in Scrum and other agile methods. It also leads to variations in the iteration length.

If you implement your task all the way into production, you get Continuous Deployment or Continuous Delivery. This puts a lot of pressure on your tooling and on automation but, I believe this is the way to go.

It is 2010, it is time to get rid of deadlines.

Scream, Project Management for the Real World

Many companies today find themselves in a situation where they have a working product, but adding new features takes forever. Even worse, when new features are added, old features stop working.

To solve this problem many companies have adapted Scrum. Scrum has a nice lightweight appeal. All you need is:

• A product owner who cares for a backlog with prioritized stories.
• A team that cares about their craft and take responsibility to deliver a subset of new features every month.
• A scrum master that makes sure the product manager and the team are playing by the rules of Scrum.

Thats it, the recipe for success...

But, what if you are not living in la-la-land where everyone on the project cares about the product?

What if your product manager doesn't keep a prioritized log of testable stories because she doesn't care. She just works here!

What if your team cares more about going surfing, than delivering well-tested, high-quality code.

Enter Scream!

Scream

Scream is project management for the real world! Scream is the way of managing unmotivated development organizations. The method itself is not new, it has been used for centuries to manage everything from husbands to entire countries.

In Scream, all you need is:

• A product manager who handle the requirements.
• A team whom will develop the requirements.
• A Scream master who will make sure that the team is playing by the rules of Scream.

At first glance, it looks deceptively like Scrum, but the rules are different: The Scream master is responsible for the product being delivered at high quality and may use any means he sees fit to make it work.

This makes all the difference in the world.

Now all you need is a Scream master with enough gust to deliver.

The Scream Master

The ideal Scream master is Begbie in Trainspotting. He has all the characteristics of a good Scream master:

Francis Begbie is an aggressive pit bull terrier, a monstrous, brawling hard man ready to explode at any moment, at anyone, for any reason. Begbie isn't afraid to test his fighting prowess against the largest of opponents. "Begbie didn't do drugs, he did people," says Renton. His sole ambition seems to be to jack someone in.

The Process

After you have selected the Scream master, you have to let him know that he will be judged on the performance of the entire development team, including, the product owner.

You also need to set up some acceptance criteria for what done is:

• All stories in the backlog, must be SMART, Specific, Measurable, Attainable, Realistic and Timely.
• The code should be DRY.
• 100% unit-test coverage of all non-trivial methods.
• Acceptance test for all stories.

The you set the project in motion.

Some Typical Scenarios

The backlog items are not SMART.

The product owner says: "I didn't have the time." Begbie: Slaps her face, "You daft c**t, these items better by SMART, right f***in' now, or I will glass you."

Bugs appear in production, due to missing unit tests.

A developer says: "It worked on my machine." Begbie: Punches him in the nose, "You f***in' buftie, if one more bug enters the system on your account you've f***in' had it."

Typical stand-up meeting:

You ken me, I'm not the type of c**t that goes looking for f***in' bother, like, but at the end of the day I'm the c**t with a pool cue and you can get the fat end in you face any time you f***ing want, like.

A scream is not only against his team and product owner, he is against everyone. This makes him excellent for dealing with impediments. Project management is all about communication and motivation and no one can get the message through like Begbie.

Begbie: "I need access to the Active Directory." SA: "I don't have the time." Begbie: "I need access to the Active Directory." SA: "You need to fill out this form." Begbie: "I NEED ACCESS TO THE ACTIVE DIRECTORY." SA: "OK, here you go."

It is my fault!

I did it! It is my fault! It was my idea that didn’t work! I fucked up!

I say the words and feel the relief flow through my body. Why? Because now the problem is under my control. One of the primary causes of stress is that the problem is beyond our control. So, just by acknowledging that the fault is mine, my stress level will lower and I will be in a better position to solve the problem. Now, when the problem is mine, I have plenty of options.

Seppuku

This will certainly solve the problem. It has unfortunate side-effects though.

Quit

If the problem is work-related, quitting will solve it. I may not get very good references from my employer, but maybe the employer is the problem in the first place. If the problem is not work-related, just decide that it is not worth the effort and quit, get divorced, throw your children out, whatever it takes.

Solve the actual problem

This is the path I usually choose, but it is good to know that the other options exist.

Programming Problems

For programming problems I usually find two gumption traps

I didn’t change anything

Sure I did, checking the status of the files, git status, quickly shows me that I made a number of changes. I can find the problem in one of those changes or I can just revert back to the previous working version and start over. A good reason for frequently checking in working code is that reverting to a previous version is less painful.

The stupid library isn’t working

Sure it is! The assertion is wrong. It should read The stupid library isn’t working as I expect it to! Now, I’m on the right track.

Why isn’t the stupid library working the way I expect it to? Because my expectations are wrong!

Did I RTFM? Yeah, kind of! All of it? Not bloody likely!

Do I know what all the things in the code means or did I just copy them from someone else? I copied some of it. Did the person I copied it from know what they were doing? I don’t know!

Whose fault is it then? Mine!

Is the stupid library really not working? No. Why am I using it? Because it solves a lot of other issues that I need to solve. Is it worth the effort? Yes? No?

Whose fault is it? Mine!

People Problems

People and process problems can be handled in a similar way. I find that many of them go away if we just realize that the problem is ours.

There is nothing wrong with Scrum, hell, there is not even anything wrong with RUP (even though I’ve written that in the past). The problem is ours.

The stupid process is not working! Sure it is! It should read the stupid process is not working as we expect it to!

The stupid process is not working as we expect it to! How do we expect it to work? Well, we don’t know. DOH!

Why is the stupid process not working as we expect it to? Because, we are not making it work as we expect it to.

Whose fault is it? Ours!

How to Handle Project Managers

There is only one thing you need to do to keep a project manager happy: Tell them how long it takes!

If you think you will be done in three hours, tell them! If you think you will be done in three days, tell them! If you don’t know if it will take between three and five days, tell them! If your original estimate does not hold, tell them! If the task is to large or fuzzy to estimate, tell them! If you don’t have a clue, get one or get yourself another job! If they think your estimates are to large, tell them to do it themselves, then, get yourself another job!

The main thing to be aware about with project managers is: They don’t produce anything of value! No matter how much they belive that their status reports and powerpoint slides are progress, they aren’t, they show progress. This is why they are happy when they know how long you will take to complete a task; it makes them believe that they are in control.

Does this mean that project managers are useless? Not at all! An invalueble project manager can make the difference between a failing project and a successful one. So, what separates a invalueble project manager from a useless one?

• An invalueble PM asks you when you will be done; a useless one tells you when you have to be done.
• An invalueble PM asks you what he can do to help; a useless one why you are not done.
• An invalueble PM shares everything; a useless one shares as little as possible.
• An invalueble PM takes responsibility for a failing project; a useless one blames the team.
• An invalueble PM gives the team credit for success; a useless one takes credit himself.
• An invalueble PM cares about the team and the project; a useless one about himself.
• An invalueble PM inspires the team; a useless one makes the team indifferent.
• An invalueble PM has a happy team; a useless one a miserable team.
• An invalueble PM knows that he is not invalueble; a useless one thinks he is.

No matter what kind of PM you have. Tell them how long it takes! works.

The Scientific Method of Programming

The scientific method is a body of techniques for investigating phenomena, acquiring new knowledge, or correcting and integrating previous knowledge. The emphasis of the method has always been on seeking truth! The method consists of a series of steps that will show that what we believe to be true is actually true. The steps are shown below:

1. Define the question.
2. Gather information and resources (observe).
3. Form hypothesis.
4. Perform experiment and collect data.
5. Analyze data.
6. Interpret data and draw conclusions that serve as a starting point for new hypotheses.
7. Publish results.
8. Retest (frequently done by other scientists).

So how does this relate to programming?

1. Identify the need.
2. Identify the task, talk to information holders, disect the problem.
3. Set up assertions and tests.
4. Write the code and run the tests.
5. Validate the tests.
6. Did the tests pass? Where the assertions correct? If not, correct it.
7. Publish results (frequently done by Cruise Control).
8. Retest (frequently done by Cruise Control).

The scientific method of programming is known by another name. It is Test Driven Development or TDD for short.

Adaptive or Rigid?

Adaptive People, Rigid Management

Adaptive People and Management

The Rise of Anarchy

At the end of the second millennium, Gregorian time, there was one major method that ruled the software industry. This method was known as the Common Rigid Artificial Process: CRAP. CRAP was a unification of three moderately successful competing methods that specialized in Object Oriented Design. The competition kept their methods from becoming overly bureaucratic.

At some point in time, the founders of the three methods, the Three Bandidos as they became known, decided that they would profit more from creating a monopoly than from competing. They assembled their methods and it became CRAP.

CRAP dominated the industry, because the three bandidos were hugely successful in both modeling and marketing, and all the major industries quickly switched to it.

Now the healthy competition between the bandidos was gone and they relished in bureaucracy. Thousands of people were using CRAP without getting anything done. If you asked anyone what they did today, they would say they produced some CRAP document that was required by the method and at the end of the day no one remembered what the goal was in the first place (to produce software). The bandidos didn‘t mind, since their goal was to make money and not software and they became wealthier and wealthier. Their future looked bright.

A New Beginning

But at an AA meeting in Utah a number of anarchists started a reactive movement against CRAP. They were fed up with producing artifacts of no value instead of real working software. They wrote the Anarchistic Manifesto and created the Anarchistic Alliance. They called it the Anarchistic Alliance so they could keep referring to it as AA in their correspondence, since they were afraid of repercussions from the CRAP affiliates.

Because they were anarchists, they could not agree on one single method that should suit all. Instead they decided to let everyone define their own method as they saw fit. The important thing was that the method had to be built on individual cooperation, instead of totalitarian hierarchies. Every individual had to take responsibility for what they were doing! No one could tell anyone else what to do! When large systems had to be produced, it had to be done through the voluntary formation of teams that would voluntarily cooperate with other teams. Informal leadership would be the rule, instead of the assigning of roles.

Before the CRAP people could do anything about it, the anarchist had infiltrated much of the small businesses that were now being targeted by CRAP. The anarchistic methods worked so well that there was no one willing to give them up in favor of CRAP. To stop the spreading of the anarchistic methods, CRAP started to make unfounded claims of how they could be used. Since they were currently only used in small and a few medium businesses, CRAP propagandist claimed that the methods would not scale. This claim was successful at first, since many people in the major industries had invested considerable amounts of time on becoming experts of the CRAP bureaucracy.

The Turning Point

Even though the propaganda worked for a while, many of the medium businesses that used CRAP went bankrupt. They did so because the software they produced was expensive and inferior to software produced by smaller but more effective firms. Soon, all the medium businesses threw CRAP out along with CRAP consultants.

At this time the large industries were still unaffected by the anarchists. They were buried too deep in bureaucracy and were also dependent on CRAP tools. This is where the CRAP people made a fundamental error. Instead of keeping up appearance and continuing to claim that anarchistic methods does not scale into large businesses, they flip-flopped. They claimed that CRAP was anarchistic! What a gargantuan mistake! Instead of making the medium industries move back to CRAP, it made the major industries realize that they had been fooled all along!

The major industries threw CRAP out as soon as they could. The expensive CRAP tools were replaced by cheaper and better alternatives and the rest, as they say, is history!

1. Anarchy, contrary to popular belief, means "absence of ruler" and not chaos.