Are you forgetting your Agile values?

A while back I wrote why sometimes Agile will fail.  In this post, I will focus on the specific misunderstandings of Agile values.   When people ask if you're Agile, they basically think:

• Do you have stand ups?
• Do you have retrospectives?
• Do you have stories?  
• Do you use yellow post its?
• etc

Such ideas belong to an Agile process called Scrum which is the most popular Agile process but isn't the only one.   There are other processes: Kanban, RUP, XP, etc...  You don't necessarily have to be Scrum.

For me the most important thing that came from Agile was the actual manifesto.  It should be read by everyone working in software at the beginning of every year and at the beginning of every project. Then it should be discussed with team members and people should be encouraged to give specific examples they have of the values and principles detailed in the manifesto.    In this blog post,  we'll have a look at the values.

8 Values with emphasis on 4

There are four points which detail 8 values in the manifesto

1. Individuals and interactions over processes and tools
2. Working software over comprehensive documentation
3. Customer collaboration over contract negotiation
4. Responding to change over following a plan

Then there is the very important sentence: "That is, while there is value on items on the right, we value items on the left more."  It is worth saying that sentence twice.  Maybe three times... four times... whatever it takes so you never forget it.  Why? Well let's take them one by one.
 

Individuals and interactions over processes and tools

So does this mean if you are Agile do you stop worrying about process?  No.  No project can ever be successful without process.   From an Agile perspective, it means, you value process. But, you value Individuals and interactions more.   So for example,  say you spend lots of man hours in a process that isn't really adding value to the project or customer. When you look at this critically it is because developers and testers don't talk to each other.  Instead they have a convoluted process so they think they can blame each other when a production defect happens.  Lots of man hours goes into this.  But, it would be much more efficient if they talked regularly to each other which then negates the aspects of the process which are making it convoluted.  So the challenge is two fold:

• Ensure people talk to each other regularly
• Ensure your processes are efficient 

So for example, it makes more sense to have a regular show and tell then no interaction whatsoever and a massive delivery at the end where you are hoping you will get customer satisfaction because of some long complex process.   Aim to make individuals, interactions in the heart of your processes.

Working software over comprehensive documentation

This is a classic misunderstanding.  Somebody wants comprehensive documentation for some complex functionality and a developer retorts: "Hey Dinasour, this isn't waterfall, there is no need for detailed documentation".   Wrong.  Documentation is still required. The point here is that with Agile you shouldn't be spending massive amounts of man hours on documentation when your software is riddled with bugs.   It is more important that your software works.  This means more time should be spend on excellent automated tests that have sufficient functional coverage.  This isn't always easy to do but you should ensure this is done. 

Customer collaboration over contract negotiation

Thirdly, do you stop doing contracts agreements with customers on your projects.  No. The point here is you spend more man hours with meetings collaborating with customers than you do teasing out a contract with lawyers.  Instead of spending a massive amount of time to get sign off on a massive project, you should collaborate thru the life cycle of the project, breaking it up into small increments, take on board feedback and work together towards a common goal: project success. 

Responding to change over following a plan

Lastly, do you stop planning?  Of course not.  But again, what is the point planning down to the nitty gritty if you cannot even respond to change? Could you imagine a customer asked for a tiny change to how a UI was displaying data and the developer team respond with: "Sorry that wasn't in our 6 month plan"? It is paramount that architecture facilitates reasonable change and if it can't the project will struggle to really embrace an agile philosophy. 

Summary

Agile is actually about putting more constraints on your software methodology.    As an analogy, the REST architecture style puts constraints on your architecture for example the Uniform Interface, Statelessness and Caching capabilities.  The idea is then by sticking to these constraints (and that's a technical challenge) you get benefits.  In the case of REST, your architecture will be more scalable, extensible and lead to much higher developer productivity for API consumers.  In the case of Agile, by sticking to the constraints of valuing 8 key concepts but putting even more emphasis on 4, your project will have greater chance of success. 

Outputting the given, when, then, Extending Spock

Spock is a Java testing framework, created in 2008 by Peter Niederwieser a software engineer with GradleWare, which facilitates amongst other things BDD.  Leveraging this example, a story may be defined as:

Story: Returns go to stock

As a store owner
In order to keep track of stock
I want to add items back to stock when they're returned.

Scenario 1: Refunded items should be returned to stock
Given that a customer previously bought a black sweater from me
And I have three black sweaters in stock.
When he returns the black sweater for a refund
Then I should have four black sweaters in stock.

Scenario 2: Replaced items should be returned to stock
Given that a customer previously bought a blue garment from me
And I have two blue garments in stock
And three black garments in stock.
When he returns the blue garment for a replacement in black
Then I should have three blue garments in stock
And three black garments in stock.

Spock makes it possible to map tests very closely to scenario specifications using the same given, when, then format. In Spock we could implement the first scenario as:

class SampleSpec extends Specification{
    def "Scenario 1: Refunded items should be returned to stock"() {
        given: "that a customer previously bought a black sweater from me"
        // ... code 
        and: "I have three black sweaters in stock."
        // ... code
        when: "he returns the black sweater for a refund"
        // ... code
        then: "I should have four black sweaters in stock."
        // ... code
    }
}

What would be nice would be to ensure accurate mapping of test scenario requirements to test scenario implementation. We could get someway down this path if we could output the syntax of the given, when, then from our test.  Spock allows us to add this functionality through its extension framework.

So, let's say our BA is really curious and wants more confidence from the developer that they stuck to the same given, when, then format and their code is in-sync. They want to get this information easily.   Developer could provide this information by first defining this annotation

import java.lang.annotation.*
import org.spockframework.runtime.extension.ExtensionAnnotation

@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.TYPE)
@ExtensionAnnotation(ReportExtension)
@interface LogScenarioDescription {}

Followed by this implementation:

import org.apache.log4j.Logger
import org.spockframework.runtime.AbstractRunListener
import org.spockframework.runtime.extension.AbstractAnnotationDrivenExtension
import org.spockframework.runtime.model.FeatureInfo
import org.spockframework.runtime.model.SpecInfo


class LogScenarioDescriptionExtension extends AbstractAnnotationDrivenExtension; {
    final static Logger logger = Logger.getLogger("scenarioLog." + ReportExtension.class);

    @Override
    void visitSpecAnnotation(Report annotation, SpecInfo spec) {
        spec.addListener(new AbstractRunListener() {
            @Override
            void afterFeature(FeatureInfo feature) {
                if (System.getEnv("logScenarios")) {
                    logger.info("***SCENARIO TEST:*** " + feature.name)
                    for (block in feature.blocks) {
                        logger.info(block.kind);
                        for (text in block.texts) {
                            logger.info(text)
                        }
                    }
                }
            }
        })
    }
}

This will then be applied to the test

@LogScenarioDescriptionExtension
class SampleSpec extends Specification{
  //...

When the test is executed it gives the following output:

***SCENARIO TEST:*** Scenario 1: Refunded items should be returned to stock
GIVEN
that a customer previously bought a black sweater from me
AND
I have three black sweaters in stock.
WHEN
he returns the black sweater for a refund
THEN
I should have four black sweaters in stock.

output to a specific logfile for scenario logging by using the following log4j:

log4j.rootLogger=INFO, stdout

log4j.logger.scenarioLog.extension.custom=INFO, scenarioLog

log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%m%n

log4j.appender.scenarioLog=org.apache.log4j.FileAppender
log4j.appender.scenarioLog.File=logs/scenario.log
log4j.appender.scenarioLog.layout=org.apache.log4j.PatternLayout
log4j.appender.scenarioLog.layout.ConversionPattern=%m%n

and now you have a logfile that your BA, QA can read! This helps foster an Agile culture of collaboration and ATDD where it possible to check that test scenarios implemented with those that were agreed.

Why Agile can fail?

Most Development teams now claim they are doing Agile and are usually doing some variant of Scrum.  In this blog post I propose three reasons why teams can struggle with Scrum or any form Agile development. And conveniently they all begin with R.

Requirements

In the pre-agile days, requirements where usually very detailed and well thought out.  For example, when I worked in Ericsson, before we moved to the Agile methodology RUP, the projects were based on a classical Waterfall model that was a very high standard (usually around CMM level 3).  The requirements came from System experts who had lots of experience in the field, were very technically skilled and had incredible domain knowledge. Their full time job was to tease things out knowing that they had effectively only one chance to get it right.

In the Agile world, with shorter iteration cycles and much more releases there are chances to make something right when you get it wrong.  Functionality can be changed around much more easily.   This is good.   It is easier for customer collaboration and thus enable more opportunities to tweak, fine tune and get all stakeholders working together sculpting the best solution.

However, because the penalty for getting requirements wrong is not as great as it is in the Waterfall model it can mean that the level of detail and clarity in requirements can start becoming insufficient and before you know development time in the sprint gets wasted trying to figure what is actually required. The key is to get the balance right.  Enough detail so that there can be clear agreement across the dev team, customer and any other stakeholders about what is coming next, but not so much detail that people are just getting bogged down in paralysis analysis and forgetting that they are supposed to be shipping regularly.

I suggest one process for helping get the balance between speed and detail for your requirements in this blog post.

Releases

In the Agile methodology Scrum you are supposed to release at the end of every sprint.  This means instead of doing  1 - 4 releases a year you will be doing closer to 20 if not more.  If your release is painful your team will struggle with any form of Agile.  For example, say a full regression test, QA, bug fixing, build, deploy etc takes 10 days (including fixing any bugs found during smoke testing) it means that 20 * 10 = 200 man days are going to releases. Whereas in the old world,  with 4 release it would just be 4 * 10 = 40 days. In case it's not obvious, that's a little bit regressive.

Now, the simple maths tells us that a team with long release cycle (for whatever reason) will struggle releasing regularly and will thus struggle with any Agile methodology.

To mitigate this risk happening, it is vital that the team has superb CI with very high code coverage and works with a very strict CI culture.  This includes:

• Ensuring developers are running full regressing tests on feature branches before merging into dev branches to minimise broken builds on dev branches
• Fix any broken build as a priority
• No checking into a broken build
• Tests are written to a high quality - they need to be as maintainable as your source code
• Coding culture where the code is written in a style so it is easily testable  (I'll cover this in a separate blog post)
• CI needs to run fast.  No point having 10,000 tests, if they take 18 hours to run. Run tests in parallel if you have to.  If your project is so massive that it really does take 18 hours to run automated tests, you need to consider some decomposition. For example, a micro-service architecture where components are in smaller and more manageable parts that can be individually released and deployed in a lot less than 18 hours.

For more information on how to achieve great CI, see here and here.

By mastering automated testing, the release cycle will be greatly shortened.  The dev team should be aiming towards a continuos delivery model where in theory any check could be released if the CI says it is green.  Now, this all sounds simple, but it is not.  In practise you need skilled developers to write good code and good tests.  But the better you are at it, the easier you will be able to release, the easier you will be able to truly agile.

Note: One of the reasons why the micro-services architectural style has become popular is because it offers an opportunity to avoid big bang releases and instead only release what needs to be. That's true.  However, most projects are not big enough or complex enough to need this approach.  Don't just jump on a bandwagon, justify every major decision.

Roles 

The most popular agile methodology Scrum only defines 3 Roles:

• Product Owner 
• Scrum Master
• Dev Team. 

That's it.  But wait sec! No Tech Lead, no Architect, no QA, no Dev manager - surely you need some of these on a project.
Of course you do. But this can be often forgotten.  Scrum is a mechanism to help manage a project, it is not a mechanism to drive quality
engineering, quality architecture and minimise technical debt.  Using Scrum is only one aspect of your process.  While your Scrum Master might governs process they don't have to govern architecture or engineering.   They may not have a clue about such matters.  If all the techies are just doing their own stories trying to complete them before the next show and tell and no-one is looking at the big picture, the project will quickly turn into an unmanageable ball of spaghetti.

This is a classical mistake at the beginning of a project. Because at the beginning there is no Tech debt. There are also no features and no bugs and of course all of this is because there is no code! But, the key point is that there is no tech debt. Everyone starts firing away at stories and there is an illusion of rapid progress but if no-one is looking at the overall big picture, the architecture, the tech debt, the application of patterns or lack of, after a few happy sprints the software entropy will very quickly explode.  Meaning that all that super high productivity in the first few weeks of the progress will quickly disappear.

To mitigate this,  I think someone technical has to back away from the coding (especially the critical path) and focus on the architecture, the technical leading, enforcing code quality. This will help ensure good design and architecture decisions are made and non-functional targets of the system are not just well defined but are met.  It is not always a glamorous job but if it ain't done, complexity will creep in and soon make everything from simple bug fixing, giving estimates, delivering new features all much harder than they should be.

In the Scrum world it is a fallacy to think every technical person must be doing a story and nothing else.  It is the equivalent of saying that everyone building a house has to be laying a brick and then wondering why the house is never finished because when the bricks never seem to line up.

Someone has to stand back ensure that people's individual work is all coming together, the tech debt is kept at acceptable levels and any technical risks are quickly mitigated.

Until the next time, take care of yourselves.

Agile Databases

Any project following an Agile methodology will usually find itself releasing to production at least 15 - 20 times per year. Even if only half of these releases involve database changes, that's 10 changes to production databases so you need a good lean process to ensure you get a good paper trail but at the same time you don't want something that that will slow you just unduly down. So, some tips in this regard:

Tip 1: Introduce a DB Log table

Use a DB Log table to capture every script run, who ran it, when it was run, what ticket it was associated with etc. Here is an example DDL for such a table for PostGres:

create sequence db_log_id_seq;
create table db_log (id int8 not null DEFAULT nextval('db_log_id_seq'), created timestamp not null,  db_owner varchar(255), db_user varchar(255), project_version varchar(255), script_link varchar(255), jira varchar(255));

W.R.T. the table columns:

• id - primary key for table. 
• timestamp - the time the script was run. This is useful.  Believe me. 
• db_owner - the user who executed the script. 
• db_user - the user who wrote the script 
• project_version_number - the version of your application / project the script was generated in.
• scrip_link - a URL link to a source controlled version of the script 
• jira - a URL to the ticket associated with the script. 

Tip 2: All Scripts should be Transactional

For every script, make sure it happens within a transaction and within the transaction make sure there is an appropriate entry into the db log table. For example, here is a script which removes a column

BEGIN;
ALTER TABLE security.platform_session DROP COLUMN IF EXISTS ttl;
INSERT INTO db_log (
       db_owner, db_user, project_version, script_link, jira, created)
VALUES (
       current_user,
       'alexstaveley',
       '1.1.4',
       'http://ldntools/labs/cp/blob/master/platform/scripts/db/updates/1.1.4/CP-643.sql',
       'CP-643',
       current_timestamp
);
COMMIT;

Tip 3: Scripts should be Idempotent

Try to make the scripts idempotent. If you have 10 developers on a team, every now and again someone will run a script twice by accident. Your db_log will tell you this, but try to ensure that when accidents happen that there is no serious damage. This means you get a simple fail safe,  rather than some newbie freaking out.   In the above script, if it is run twice the answer will be the exact same.

Tip 4: Source Control your Schema

Source control a master DDL for the entire project. This is updated anytime the schema changes. Meaning you have update scripts and a complete master script containing the DDL for entire project. The master script is run at the beginning of every CI, meaning that:

• Your CI always starts with a clean database 
• If a developer forgets to upgrade the master script, the CI will fail and your team will quickly know the master script needs to be updated.
• When you have a master script it gives you two clear advantages: 
• New developers get up and running with a clean database very quickly
• It becomes very easy to provision new environments. Just run the master script! 

Tip 5: Be Dev Friendly

Make it easy for developers to generate the master script. Otherwise when the heat is on, it won't get done.

Tip 6: Upgrade and Revert

For every upgrade script write a corresponding revert script. Something unexpected happens in production, you gotta be able to reverse the truck back out!

BEGIN;

ALTER TABLE security.platform_session ADD COLUMN hard_ttl INT4;
UPDATE security.platform_session  SET hard_ttl = -1 WHERE hard_ttl IS NULL;
ALTER TABLE security.platform_session ALTER COLUMN hard_ttl SET NOT NULL;

ALTER TABLE security.platform_session ADD COLUMN ttl INT4;
UPDATE security.platform_session  SET ttl = -1 WHERE ttl IS NULL;
ALTER TABLE security.platform_session ALTER COLUMN ttl SET NOT NULL;


INSERT INTO db_log (
       db_owner, db_user, platform_version, script_link, jira, created)
       values (
       current_user,
       'alexstaveley',
       '1.1.4',
       'http://ldntools/labs/cp/blob/master/platform/scripts/db/reverts/1.1.4/revert-CP-463.sql',
       'CP-463',
       current_timestamp
    );

COMMIT;

Until the next time take care of yourselves.