Wednesday, June 8, 2016

Triggering a Client Cache Refresh

More and more web sites are using a single page style architecture.  This means there is a bunch of static resources (aka assets) including:
  • JS 
  • CSS
  • HTML templates
  • Images
  • ...
all residing in the client's browser.  For performance reasons, the static content is usually cached. And like all caches, eventually the data gets stale and the cache must be refreshed.  One excellent technique for achieving this in the web tier is Cache busting (see: here and here).
However, even using this excellent pattern there still needs to be some sort of trigger to indicate that things have gone out of date.  Some options:

HTML5 Sockets

In this case a server can send a request to any client to say go and get the new stuff.

Comet style polling

The Client consistently polls the server in the background asking if there is a new version available. When the server says there is, the client is triggered to start cache busting.

Both of these are good approaches but in some edge cases may not be available:
  1. Architecture may not allow HTML5 sockets - might be some secure banking web site that just doesn't like it.
  2. The Comet style polling might be too intensive and may just leave open edge cases where a critical update is needed but the polling thread is waiting to get fired. 

Client refresh notification pattern

Recently, on a project neither of the above approaches were available, so I needed something else which I shall now detail.  The solution was based on some existing concepts already in the architecture (which are useful for other things) and some new ones that needed to be introduced:
  1. The UI always knows the current version it is on.  This was already burnt into the UI as part of the build process.
  2. Every UI was already sending up the version it is on in a custom header. It was doing this for every request.  Note: this is very useful as it can make diagnosing problems easier. Someone seeing weird problems, nice to be able see their are on a old version straight away.
The following new concepts were then introduced:
  1. The server would now always store the latest client version it has received.  This is easy to do and again handy thing to have anyway.
  2. The server would then always add a custom header to every response to indicate the latest client version it has received.  Again, useful thing to have for anyone doing a bit of debugging with firebug or chrome web tools
  3. Some simple logic would then be in the client, so that when it saw a different version in response to the one it sent up, it knows there's a later version out there and that's the trigger to start cache busting!   This should be done in central place for example an Angular filter (if you are using Angular)
  4. Then as part of every release, just hit the server with latest client in the smoke testing. 
  5. As soon as any other client makes a request, it will be told that there is a later client version out there and it should start cache busting.

So the clients effectively tell each other about the latest version but without ever talking to each other it's all via the server.  It's like the mediator pattern. But it's still probably confusing. So let's take a look at a diagram.



With respect to diagram above: 
  • UI 1 has a later version (for example 1.5.1) of static assets than UI 2  (for example 1.5.0)
  • Server thinks the latest version static assets is 1.5.0
  • UI 1 then makes a request to the server and sends up the version of static assets it has e.g. 1.5.1
  • Server sees 1.5.1 is newer than 1.5.0 and then updates its latest client version variable to 1.5.1
  • UI 2 makes a request to the server and sends up the version of static assets it has which is 1.5.0
  • Server sends response to UI 2 with a response header saying that the latest client version is 1.5.1
  • UI 2 checks and sees that the response header client version is different to the version sent up and then starts busting the cache
Before the observant amongst you start saying this will never work in a real enterprise environment as you never have just one server (as in one JVM) you have several - true.  But then you just store the latest client version in a distributed cache (e.g. Infinispan) that they can all access. 

Note: In this example I am using two web clients and one back end server.  But note the exact same pattern could be used for back end micro-services that communicate with each other.  Basically anything where there is a range of distributed clients (they don't have to be web browsers) and caching of static resources is required this could be used. 

Until the next time, take care of yourselves. 


Tuesday, June 7, 2016

Why Agile can fail?

Most Development teams now claim they are doing Agile and are usually doing some variant of Scrum.  In this blog post I propose three reasons why teams can struggle with Scrum or any form Agile development. And conveniently they all begin with R.

Requirements

In the pre-agile days, requirements where usually very detailed and well thought out.  For example, when I worked in Ericsson, before we moved to the Agile methodology RUP, the projects were based on a classical Waterfall model that was a very high standard (usually around CMM level 3).  The requirements came from System experts who had lots of experience in the field, were very technically skilled and had incredible domain knowledge. Their full time job was to tease things out knowing that they had effectively only one chance to get it right.

In the Agile world, with shorter iteration cycles and much more releases there are chances to make something right when you get it wrong.  Functionality can be changed around much more easily.   This is good.   It is easier for customer collaboration and thus enable more opportunities to tweak, fine tune and get all stakeholders working together sculpting the best solution.

However, because the penalty for getting requirements wrong is not as great as it is in the Waterfall model it can mean that the level of detail and clarity in requirements can start becoming insufficient and before you know development time in the sprint gets wasted trying to figure what is actually required. The key is to get the balance right.  Enough detail so that there can be clear agreement across the dev team, customer and any other stakeholders about what is coming next, but not so much detail that people are just getting bogged down in paralysis analysis and forgetting that they are supposed to be shipping regularly.

I suggest one process for helping get the balance between speed and detail for your requirements in this blog post.

Releases

In the Agile methodology Scrum you are supposed to release at the end of every sprint.  This means instead of doing  1 - 4 releases a year you will be doing closer to 20 if not more.  If your release is painful your team will struggle with any form of Agile.  For example, say a full regression test, QA, bug fixing, build, deploy etc takes 10 days (including fixing any bugs found during smoke testing) it means that 20 * 10 = 200 man days are going to releases. Whereas in the old world,  with 4 release it would just be 4 * 10 = 40 days. In case it's not obvious, that's a little bit regressive.

Now, the simple maths tells us that a team with long release cycle (for whatever reason) will struggle releasing regularly and will thus struggle with any Agile methodology.

To mitigate this risk happening, it is vital that the team has superb CI with very high code coverage and works with a very strict CI culture.  This includes:
  • Ensuring developers are running full regressing tests on feature branches before merging into dev branches to minimise broken builds on dev branches
  • Fix any broken build as a priority
  • No checking into a broken build
  • Tests are written to a high quality - they need to be as maintainable as your source code
  • Coding culture where the code is written in a style so it is easily testable  (I'll cover this in a separate blog post)
  • CI needs to run fast.  No point having 10,000 tests, if they take 18 hours to run. Run tests in parallel if you have to.  If your project is so massive that it really does take 18 hours to run automated tests, you need to consider some decomposition. For example, a micro-service architecture where components are in smaller and more manageable parts that can be individually released and deployed in a lot less than 18 hours.
For more information on how to achieve great CI, see here and here.

By mastering automated testing, the release cycle will be greatly shortened.  The dev team should be aiming towards a continuos delivery model where in theory any check could be released if the CI says it is green.  Now, this all sounds simple, but it is not.  In practise you need skilled developers to write good code and good tests.  But the better you are at it, the easier you will be able to release, the easier you will be able to truly agile.

Note: One of the reasons why the micro-services architectural style has become popular is because it offers an opportunity to avoid big bang releases and instead only release what needs to be. That's true.  However, most projects are not big enough or complex enough to need this approach.  Don't just jump on a bandwagon, justify every major decision.

Roles 

The most popular agile methodology Scrum only defines 3 Roles:
  • Product Owner 
  • Scrum Master
  • Dev Team. 
That's it.  But wait sec! No Tech Lead, no Architect, no QA, no Dev manager - surely you need some of these on a project.
Of course you do. But this can be often forgotten.  Scrum is a mechanism to help manage a project, it is not a mechanism to drive quality
engineering, quality architecture and minimise technical debt.  Using Scrum is only one aspect of your process.  While your Scrum Master might governs process they don't have to govern architecture or engineering.   They may not have a clue about such matters.  If all the techies are just doing their own stories trying to complete them before the next show and tell and no-one is looking at the big picture, the project will quickly turn into an unmanageable ball of spaghetti.

This is a classical mistake at the beginning of a project. Because at the beginning there is no Tech debt. There are also no features and no bugs and of course all of this is because there is no code! But, the key point is that there is no tech debt. Everyone starts firing away at stories and there is an illusion of rapid progress but if no-one is looking at the overall big picture, the architecture, the tech debt, the application of patterns or lack of, after a few happy sprints the software entropy will very quickly explode.  Meaning that all that super high productivity in the first few weeks of the progress will quickly disappear.

To mitigate this,  I think someone technical has to back away from the coding (especially the critical path) and focus on the architecture, the technical leading, enforcing code quality. This will help ensure good design and architecture decisions are made and non-functional targets of the system are not just well defined but are met.  It is not always a glamorous job but if it ain't done, complexity will creep in and soon make everything from simple bug fixing, giving estimates, delivering new features all much harder than they should be.

In the Scrum world it is a fallacy to think every technical person must be doing a story and nothing else.  It is the equivalent of saying that everyone building a house has to be laying a brick and then wondering why the house is never finished because when the bricks never seem to line up.



Someone has to stand back ensure that people's individual work is all coming together, the tech debt is kept at acceptable levels and any technical risks are quickly mitigated.

Until the next time, take care of yourselves.




Thursday, June 2, 2016

Agile Databases

Any project following an Agile methodology will usually find itself releasing to production at least 15 - 20 times per year. Even if only half of these releases involve database changes, that's 10 changes to production databases so you need a good lean process to ensure you get a good paper trail but at the same time you don't want something that that will slow you just unduly down. So, some tips in this regard:

Tip 1: Introduce a DB Log table

Use a DB Log table to capture every script run, who ran it, when it was run, what ticket it was associated with etc. Here is an example DDL for such a table for PostGres:
create sequence db_log_id_seq;
create table db_log (id int8 not null DEFAULT nextval('db_log_id_seq'), created timestamp not null,  db_owner varchar(255), db_user varchar(255), project_version varchar(255), script_link varchar(255), jira varchar(255));
W.R.T. the table columns:
  • id - primary key for table. 
  • timestamp - the time the script was run. This is useful.  Believe me. 
  • db_owner - the user who executed the script. 
  • db_user - the user who wrote the script 
  • project_version_number - the version of your application / project the script was generated in.
  • scrip_link - a URL link to a source controlled version of the script 
  • jira - a URL to the ticket associated with the script. 

Tip 2: All Scripts should be Transactional

For every script, make sure it happens within a transaction and within the transaction make sure there is an appropriate entry into the db log table. For example, here is a script which removes a column
BEGIN;
ALTER TABLE security.platform_session DROP COLUMN IF EXISTS ttl;
INSERT INTO db_log (
       db_owner, db_user, project_version, script_link, jira, created)
VALUES (
       current_user,
       'alexstaveley',
       '1.1.4',
       'http://ldntools/labs/cp/blob/master/platform/scripts/db/updates/1.1.4/CP-643.sql',
       'CP-643',
       current_timestamp
);
COMMIT;

Tip 3: Scripts should be Idempotent

Try to make the scripts idempotent. If you have 10 developers on a team, every now and again someone will run a script twice by accident. Your db_log will tell you this, but try to ensure that when accidents happen that there is no serious damage. This means you get a simple fail safe,  rather than some newbie freaking out.   In the above script, if it is run twice the answer will be the exact same.

Tip 4: Source Control your Schema

Source control a master DDL for the entire project. This is updated anytime the schema changes. Meaning you have update scripts and a complete master script containing the DDL for entire project. The master script is run at the beginning of every CI, meaning that:
  • Your CI always starts with a clean database 
  • If a developer forgets to upgrade the master script, the CI will fail and your team will quickly know the master script needs to be updated.
  • When you have a master script it gives you two clear advantages: 
    • New developers get up and running with a clean database very quickly
    • It becomes very easy to provision new environments. Just run the master script! 

Tip 5: Be Dev Friendly

Make it easy for developers to generate the master script. Otherwise when the heat is on, it won't get done.

Tip 6: Upgrade and Revert

For every upgrade script write a corresponding revert script. Something unexpected happens in production, you gotta be able to reverse the truck back out!
BEGIN;

ALTER TABLE security.platform_session ADD COLUMN hard_ttl INT4;
UPDATE security.platform_session  SET hard_ttl = -1 WHERE hard_ttl IS NULL;
ALTER TABLE security.platform_session ALTER COLUMN hard_ttl SET NOT NULL;

ALTER TABLE security.platform_session ADD COLUMN ttl INT4;
UPDATE security.platform_session  SET ttl = -1 WHERE ttl IS NULL;
ALTER TABLE security.platform_session ALTER COLUMN ttl SET NOT NULL;


INSERT INTO db_log (
       db_owner, db_user, platform_version, script_link, jira, created)
       values (
       current_user,
       'alexstaveley',
       '1.1.4',
       'http://ldntools/labs/cp/blob/master/platform/scripts/db/reverts/1.1.4/revert-CP-463.sql',
       'CP-463',
       current_timestamp
    );

COMMIT;

Until the next time take care of yourselves.