tag:blogger.com,1999:blog-89657342772834659442024-02-21T21:28:38.903-08:00Dublin TechSome simple bits and pieces about Software Architecture. Generally, a JVM is somewhere...Alex Staveleyhttp://www.blogger.com/profile/04378245339254378938noreply@blogger.comBlogger102125tag:blogger.com,1999:blog-8965734277283465944.post-26438558564091027552020-03-28T08:32:00.010-07:002023-02-26T13:41:08.645-08:00Clean Unit Testing<div dir="ltr" style="text-align: left;" trbidi="on">
It's easy to write "unit test" tests that use JUnit and some mocking library. They may produce code coverage, that keep some stakeholders happy, even though the tests aren't even unit tests and provide questionable value. It can also be very easy to write unit tests that are — in theory — units test but are more complex than the underlying code and hence just add to the total software entropy.
<br />
<br />
This particular type of software entropy has the unpleasant characteristic of making it even harder for the underlying software to be restructured or to surface new requirements. It is like the test has a negative value.
<br />
<br />
Doing unit testing properly is a lot harder than people think. In this article, I outline several tips that aim to improve the readability, maintainability and the quality of your unit tests.
<br />
<br />
Note: for the code snippets, Spock is used. For those who don't know Spock, consider it a very powerful DSL around JUnit which adds some nice features and cuts down on verbosity.
<br />
<br />
<h3 style="text-align: left;">
Reason for Failure </h3>
<div>
<br /></div>
The Unit Test should only fail if there is a problem with the code under test. A unit test for the class DBService should only fails if there is a bug with DBService not if there is a bug with any other class it depends on. Thus, in the unit test for DBService, the only instantiated object should be DBService. Every other object that DBService depends on should be stubbed or mocked.
<br />
Otherwise, you are testing code beyond DBService. While you might incorrectly think this is more bang for the buck, it means locating the root cause of problems will take longer. If the test fails, it could be because there is a problem with multiple classes but you don't know which one. Whereas, if it can only fail because the code under test is wrong, then you know exactly where the problem is.
<br />
Furthermore, thinking this way will improve the Object Orientated nature of your code. The tests will only test the responsibilities of the Class. If it's responsibilities aren't clear, or it can't do anything without another class, or the class is so trivial the test is pointless, it prompts the question that there something wrong with the class in terms of its responsibilities.
<br />
The only exception to not mocking or stubbing a dependent class is if you are using a well-known class from the Java library e.g. String. There is not much point stubbing or mocking that. Or, the dependent class is just a simple immutable POJO where there is not much value to stubbing or mocking it.
<br />
<br />
<h3 style="text-align: left;">
Stubbing and Mocking </h3>
<div>
<br /></div>
The terms mocking and stubbing can often be used interchangeably as if there were the same thing. They are <b>not </b>the same thing. In summary, if your code under test has a dependency on an object for which it never invokes a method on that object which has side effects, that object should be stubbed.
<br />
Whereas, if it has a dependency on an object for which it does invoke methods that have side effects then that should be mocked. Why is this important? Because your test should be checking for different things depending on the types of relationships it has with its dependencies.
<br />
Let's say your object under test is BusinessDelegate. BusinessDelegate receives requests to edit BusinessEntities. It performs some simple business logic and then invokes methods on a DBFacade (a facade class in front of a Database). So, the code under test looks like this:
<br />
<pre>
<code class="language-java">public class BusinessDelegate {
private DBFacade dbFacade;
// ...
public void edit(BusinessEntity businessEntity) {
// Read some attributes on the business entity
String newValue = businessEntity.getValue();
// Some Business Logic, Data Mapping, and / or Validation
//...
dbFacade.update(index, data)
}
}
</code>
</pre>
Regarding the BusinessDelegate class, we can see two relationships.<br />
<br />
<br />
<ol>
<li>A read-only relationship with BusinessEntity. The BusinessDelegate calls a few getters() on it and never changes its state or never invokes any methods that have side effects. </li>
<li>A relationship with DBFacade where it asks DBFacade to do something that we assume will have side effects. It is not the responsibility of BusinessDelegate to ensure the update happens, that is DBFacade's job. The responsibility of BusinessDelegate is to ensure the update method is invoked with the correct parameters — only.</li>
</ol>
So clearly, regarding the unit test for BusinessDelegate, BusinessEntity should be stubbed and DbFacade should be mocked. If we were using the Spock testing framework we could see this very clearly<br />
<pre>
<code class="groovy">class BusinessDelegateSpec {
@Subject
BusinessDelegate businessDelegate
def dbFacade
def setup() {
dbFacade = Mock(DbFacade)
businessDelegate = new BusinessDelegate(dbFacade);
}
def "edit(BusinessEntity businessEntity)"() {
given:
def businessEntity = Stub(BusinessEntity)
// ...
when:
businessDelegate.edit(businessEntity)
then:
1 * dbFacade.update(data)
}
}
</code>
</pre>
Having a good understanding of stub mock differentiation improves OO quality dramatically. Instead of just thinking about what the object does, the relationships and dependencies between them get much more focus. It is now possible for unit tests to help enforce design principles that would otherwise just get lost.<br />
<div>
<br />
<h4 style="text-align: left;">
Stub and Mock in the Right Place </h4>
<div>
<br /></div>
The curious among you, might be wondering why in the above code sample, dbFacade declared at class level, while businessEntity was declared at method level? Well, the answer is, unit test code is much more readable the more it can mirror the code under test. In the actual BusinessDelegate class, the dependency on dbFacade is at the class level and the dependency on BusinessEntity at the method level.
<br />
In the real world when a BusinessDelegate is instantiated a DbFacade dependency will exist, anytime BusinessDelegate is instantiated for a unit test it is ok to have the DbFacade dependency also existing.
<br />
Sound reasonable? Hope so. There are two further advantages of doing this:
<br />
<ul>
<li>A reduction in code verbosity. Even using Spock, unit tests can become verbose. If you move class-level dependencies out of the unit test, you will reduce test code verbosity. If your class has a dependency on four other classes at the class level that minimum four lines of code out of each test.
</li>
<li>Consistency. Developers tend to write unit tests their way. Fine if they are the only people reading their code; but this is rarely the case. Therefore, the more consistency we have across the tests the easier they are to maintain. So if you read a test you have never read before and at least see variables being stubbed and mocked in specific places for specific reasons, you will find the unit test code easier to read.</li>
</ul>
<div>
<br /></div>
<h3 style="text-align: left;">
Variable Declaration Order </h3>
<div>
<br /></div>
<div>
This is a follow on from the last point. Declaring the variables in the right place is a great start, the next thing is to do in the same order they appear in the code. So, if we have something like below.
<br />
<pre>
<code class="language-java">public class BusinessDelegate {
private BusinessEntityValidator businessEntityValidator;
private DbFacade dbFacade;
private ExcepctionHandler exceptionHandler;
@Inject
BusinessDelegate(BusinessEntityValidator businessEntityValidator, DbFacade dbFacade, ExcepctionHandler exceptionHandler) {
// ...
// ...
}
public BusinessEntity read(Request request, Key key) {
// ...
}
}
</code>
</pre>
It is much easier to read the test code if they stubs and mocks are defined in the same order as the way the class declares them.
<br />
<pre>
<code class="groovy">class BusinessDelegateSpec {
@Subject BusinessDelegate businessDelegate
// class level dependencies in the same order
def businessEntityValidator
def dbFacade
def exceptionHandler
def setup() {
businessEntityValidator = Stub(BusinessEntityValidator)
dbFacade = Mock(DbFacade)
exceptionHandler = Mock(ExceptionHandler)
businessDelegate = new BusinessDelegate(businessEntityValidator, dbFacade, exceptionHandler)
}
def "read(Request request, Key key)"() {
given:
def request = Stub(Request)
def key = Stub(key)
when:
businessDelegate.read(request, key)
then:
// ...
}
}
</code>
</pre>
<h3 style="text-align: left;">
</h3>
<h3 style="text-align: left;">
Variable Naming </h3>
<div>
<br /></div>
<div>
And if you thought the last point was pedantic, you'll be glad to know this one also is. The variable names used to represent the stubs and mocks should be the same names that are used in the actual code.
Even better, if you can name the variable the same as the type in the code under test and not lose any business meaning then do that. In the last code sample, the parameter variables are named requestInfo and key and they corresponding stubs have the same names. This is much easier to follow than doing something like this:
<br />
<pre><code class="language-java">//..
public void read(Request info, Key someKey) {
// ...
}
</code>
</pre>
<pre>
<code class="groovy">
// corresponding test code
def "read(Request request, Key key)"() {
given:
def aRequest = Stub(Request)
def myKey = Stub(key) // you ill get dizzy soon!
// ...
</code>
</pre>
<h3 style="text-align: left;">
</h3>
<h3 style="text-align: left;">
Avoid Over Stubbing </h3>
</div>
<div>
<br /></div>
<div>
Too much stubbing (or mocking) usually means something has gone wrong. Let's consider the Law of Demeter. Imagine some telescopic method call...
<br />
<pre>
<code class="language-java">
List<businessentity> queryBusinessEntities(Request request, Params params) {
// check params are allowed
Params paramsToUpdate = queryService.getParamResolver().getParamMapper().getParamComparator().compareParams(params)
// ...
// ...
}
</businessentity>
</code>
</pre>
It is not enough to stub queryService. Now whatever is returned by resolveAllowableParams() has to be stubbed and that stub has to have mapToBusinessParamsstubbed() which then has to have mapToComparableParams() stubbed. Even with a nice framework like Spock which minimizes verbosity, you will have to four lines of stubbing for what is one line of Java code.
<br />
<pre>
<code class="groovy">def "queryBusinessEntities()"() {
given:
def params = Stub(Params)
def paramResolver = Stub(ParamResolver)
queryService.getParamResolver() = paramResolver
def paramMapper = Stub(ParamMapper)
paramResolver.getParamMapper() >> paramMapper
def paramComparator = Stub (ParamComparator)
paramMapper.getParamComparator() >> paramComparator
Params paramsToUpdate = Stub(Params)
paramComparator.comparaParams(params) >> paramsToUpdate
when:
// ...
then:
// ...
}
</code>
</pre>
Yuck! Look at how that one line of Java does to our unit test. It gets even worse if you are not using something like Spock. The solution is to avoid telescopic method calling and try to just use direct dependencies.
In this case, just inject theParamComparator directly into our class. Then the code becomes...
<br />
<pre>
<code class="language-java">
List<businessentity> queryBusinessEntities(Request request, Params params) {
// check params are allowed
Params paramsToUpdate = paramComparator.compareParams(params)
// ...
// ...
}
</businessentity>
</code></pre>
and the test code becomes
<br />
<pre>
<code class="groovy">
setup() {
// ...
// ...
paramComparator = Stub (ParamComparator)
businessEntityDelegate = BusinessEntityDelegate(paramComparator)
}
def "queryBusinessEntities()"() {
given:
def params = Stub(Params)
Params paramsToUpdate = Stub(Params)
paramComparator.comparaParams(params) >> paramsToUpdate
when:
// ..
then:
// ...
}
</code>
</pre>
All of the sudden people should be thanking you for feeling less dizzy. </div>
<div>
<br /></div>
<h3 style="text-align: left;">
Gherkin Syntax </h3>
<div>
<br /></div>
<div>
Bad unit tests have horrible things like asserts all over the place The top the middle and the bottom. It can very quickly get nauseating trying to figure out which ones are important, which ones are redundant and which ones require which bit of set up etc etc.
Schematic things are easier to follow. That is the real advantage of the Gherkin syntax. The scenario is set up in the <b>given:</b> <i>always</i>, the <b>when: </b>is the test scenario and <b>then: </b>is what we expect. Even better using, something like Spock means you have a nice, neat DSL so that the <i>given:, when:, and then:</i> can all be co-located in the one test method.</div>
<h3 style="text-align: left;">
</h3>
<h3 style="text-align: left;">
Narrow When Wide Then </h3>
<div>
If a unit test is testing four methods, is it a unit test? Consider the below test:
<br />
<pre>
<code class="groovy">
def "test several methods" {
given:
// ...
when:
def name = personService.getname();
def dateOfBirth = personService.getDateOfBirth();
def country = personService.getCountry();
then:
name == "tony"
dateOfBirth == "1970-04-04"
country == "Ireland"
}
</code>
</pre>
First up, if Jenkins tells you this failed, you are going to have to root around and figure out what part of the class is wrong. Because the test doesn't focus on a specific method you don't know immediately which method is failing. Second up, say if it is getName() that is failing, how do you know getDateOfBirth() and getCountry() are working?
The test stops on the first failure. So when the test fails, you don't even know if you have one method not working or three methods not working. You can go around telling everyone you have 99% code coverage and one test failing. But — how much was that one test really doing?
<br />
Furthermore, what's easier to fix? A small test or a long test? Ideally, a test should check a single interaction with the thing you are testing. Now, this doesn't mean you can only have one asset, but you should have a narrow when and a wide then.
<br />
So let's take the narrow when first. Ideally, one line of code only. The one line of code matches the method you are unit testing.
<br />
<pre>
<code class="groovy">
def "getName()" {
given:
// ...
when:
def name = personService.getname();
then:
name == "tony"
}
def "getDateOfBirth()" {
given:
// ...
when:
def dateOfBirth = personService.getDateOfBirth();
then:
dateOfBirth == "1970-04-04"
}
def "getCountry()" {
given:
// ...
when:
def country = personService.getCountry();
then:
country == "Ireland"
}
</code>
</pre>
Now we could have the exact same code coverage, if getName() fails but getCountry() and getDateOfBirth() pass, but there is a problem with getName() and not getCountry() and getDateOfBirth(). Getting the granularity of a test is an entirely different stat to code coverage. It should be ideally one unit test minimum for every non-private method. It is more when you factor in negative tests etc.
It is perfectly fine to have multiple asserts in a unit test. For example, suppose we had a method that delegated onto other classes.
<br />
Consider a method resyncCache() which in its implementation calls two other methods on a cacheService object, clear() and reload().
<br />
<pre>
<code class="language-groovy">
def "resyncCache()" {
given:
// ...
when:
personService.resyncCache();
then:
1 * cacheService.clear()
1 * cacheService.reload()
}
</code>
</pre>
In this scenario, it would not make sense to have two separate tests. The "when" is the same and if either fails, you know immediately which method you have to look at. Having two separate tests just means twice the effort with little benefit. The subtle thing to get right here is to ensure your assets are in the right order. They should be in the same order as code execution.
So, clear() is invoked before reload(). If the test fails at clear(), there is not much point going on to check to reload() anyway as the method is broken. If you don't follow the assertion order tip, and assert on reload() first and that is reported as failing, you won't know if clear() invocation which is supposed to happen first even happened. Thinking this way will make help you become a Test Ninja!
<br />
The ordering tip for mocking and stubbing, the same applies to assert. Assert in chronological order. It's pedantic but it will make test code much more maintainable.<br />
<h3 style="text-align: left;">
Parameterization </h3>
The parameterization is a very powerful capability that can greatly reduce test code verbosity and rapidly increase branch coverage in code paths. The Unit Test Ninja should be always able to spot when to use it!
<br />
An obvious indication that a number of tests could be grouped into one test and parameterized is that they have the same when blocks, except for different input parameters.
<br />
For example, consider the below.
<br />
<pre>
<code class="groovy">
def "addNumbers(), even numbers"() {
given:
// ...
when:
def answer = mathService.addNumbers(4, 4);
then:
// ...
}
def "addNumbers(), odd numbers"() {
given:
// ...
when:
def answer = mathService.addNumbers(5, 5);
then:
// ...
}
</code>
</pre>
As we can see here the when is the same except the input parameters. This is a no-brainer for parameterization.
<br />
<pre>
<code class="groovy">
@Unroll("number1=#number1, number2=#number2") // unroll will provide the exact values in test report
def "addNumbers()"(int number1, int number2) {
given:
// ...
when:
def answer = mathService.addNumbers(number1, number2);
then:
// ...
where:
number1 | number2 || answer
4 | 4 || 8
5 | 5 || 10
}
</code>
</pre>
Immediately we get a 50% reduction in code. We have also made it much easier to add further permutations by just adding another row to the where table.
So, while it may seem very obvious that these two tests should have been the one parameterized test, it is only obvious if the maxim of having a narrow when is adhered to. The narrow "when" coding style makes the exact scenario being tested much easier to see. If a wide when is used with lots of things happening it is not and therefore spotting tests to parameterize is harder.
<br />
Usually, the only time to not parameterize a test that has the same syntactic <b>where: code block</b> is when the expectations are a completely different structure. Expecting an int is the same structure, expecting an exception in one scenario and an int in another means you have two different structures. In such scenarios, it is better not to parameterize. A proverbial (and infamous) example of this is mixing a positive and negative test.
<br />
Suppose our addNumbers() method will throw an exception if it receives a float, that's a negative test and should be kept separate. <b>A then: block</b> should never contain an <i>if statement</i>. It is a sign a test is becoming too flexible and a separate test with <i>no if statements </i>would make more sense.
<br />
<h3 style="text-align: left;">
</h3>
<h3 style="text-align: left;">
Summary </h3>
Clean unit testing is very important if you want to have maintainable code, release regularly and enjoy your Software Engineering more.
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br /></div>
</div>
</div>
</div>
Alex Staveleyhttp://www.blogger.com/profile/04378245339254378938noreply@blogger.com0tag:blogger.com,1999:blog-8965734277283465944.post-54682597130587054562019-12-08T10:29:00.001-08:002020-04-25T10:11:44.240-07:00Arch Unit<div dir="ltr" style="text-align: left;" trbidi="on">
If Software Architecture is done to a reasonable standard, we should expect to see:
<br />
<ul>
<li>Well designed patterns that can fulfill both functional requirements and non-functional requirements
</li>
<li>No crazy crazy coupling, concerns are properly separated and everything is testable.
</li>
</ul>
If we get that, we should have confidence that as the software evolves it is maintainable. So the tricky part is all too often Architectural rules start off great on a whiteboard (or a powerpoint slide) but just get lost in code, because they are too difficult to enforce.
<br />
<br />
<a href="https://www.archunit.org/">Arch Unit</a> is a super mechanism to impose Architectural rules and patterns on your code base. It has been around a few years but something I only discovered this year. I came across it when I was trying to think of ways to ensure the proverbial "utils" package did not turn into the proverbial "d<i>umping ground</i>". Ideally, we'd have no utils packages ever. But, in the real world they nearly always exist. The utils package shouldn't have many <a href="https://en.wikipedia.org/wiki/Efferent_coupling">efferent dependencies</a>. So for example, suppose you have a package called <i>shoppingcart</i>. Then you have need some sort of utility function to add the total of two carts, remove special offers, add loyalty discounts, blah blah blah. The last thing you want to see is someone checking that into the <i>utils</i> package with dependencies towards the shoppingcart package. Because, if it is so shoppingcart focused, it should really just be in the shoppingcart package. If this happens, very soon your utils package will have dependencies to everything and everything will have dependencies to it. Disaster. What is the point in packages if anything can just depend on anything? They will cease to provide any name-spacing or encapsulation benefits.
<br />
<br />
So, how can Arch Unit help? Well very simple you define Architectural rules like a JUnit test. Wait a sec... It is a JUnit test. The <i>efferent </i>(outward) and <i>afferent </i>(inward) for you utils package are very simple expressed as:
<br />
<br />
<pre class="brush: java; ruler: true; first-line: 1">@ArchTest
public static final ArchRule utilPackageDependenciesRules = classes().that().resideInAPackage("com.company.application.util")
.should().onlyDependOnClassesThat().resideInAnyPackage(getAllowedDependencies("com.company.application.exception"))
.andShould().onlyHaveDependentClassesThat().resideInAnyPackage("com.company.application.shoppingcart"
"com.company.application.payment);
</pre>
So that's it. Now repeat for every package and you now have code control that runs like any other JUnit test. So therefore it will run easily as part of your CI, CD etc.
Now, if you have architected your packages well, you don't have to bring up at code reviews. Instead the rules are part of your CI.
As your software evolves and new packages come along and dependencies rules change, simply just change the rules that are expressed in nice fluent Java APIs.
Someone new joins the teams and wants to get up to speed on the Architectural package rules? Simple, just direct them Architectural tests.
<br />
<br />
Not only does ArchUnit give you the ability to express package rules, you can also define your own rules aka <i>conditions</i> and then apply them to whatever code you want.
For example, suppose you want a condition that an object is immutable. You naturally therefore want no setters. That could be expressed by this condition.
<br />
<pre class="brush: java; ruler: true; first-line: 1"> static ArchCondition<javaclass> noPublicSettersCondition =
new ArchCondition<javaclass>("class has no public setters") {
@Override
public void check(JavaClass item, ConditionEvents events) {
for (JavaMethod javaMethod: item.getMethods()) {
if (javaMethod.getName().startsWith("set") &&
javaMethod.getModifiers().contains(JavaModifier.PUBLIC)) {
String message = String.format(
"Public method %s is not allowed begin with setter", javaMethod.getName());
events.add(SimpleConditionEvent.violated(item, message));
}
}
}
};
</javaclass></javaclass></pre>
You could then apply the noSetter condition to any custom Exception a developer may write. It wouldn't be good if an Exception had a setter would it?
<br />
<pre> @ArchTest
public static final ArchRule noExceptionsHaveSetters = classes().that()
.areAssignableTo(RuntimeException.class).should(noSettersCondition);
</pre>
Suppose you keep noticing that Loggers defined in classes either <i>aren't private, aren't static or aren't final</i>. Don't waste time talking about it code reviews. ArchUnit it!
<br />
<pre> @ArchTest
public final ArchRule loggers_should_be_private_static_final =
fields().that().haveRawType(TaLogger.class)
.should().bePrivate()
.andShould().beStatic()
.andShould().beFinal()
.because("we agreed on this convention");
</pre>
So the goal here is to conceptualise good rules that will help your to remain testable and maintainable and then enforce them in a way that is easy to check and understand.
ArchUnit really is a great library tool.
</div>
Alex Staveleyhttp://www.blogger.com/profile/04378245339254378938noreply@blogger.com0tag:blogger.com,1999:blog-8965734277283465944.post-65554781068539653072019-06-09T09:31:00.000-07:002020-04-25T10:12:13.550-07:00Defining a Resource <div dir="ltr" style="text-align: left;" trbidi="on">
<a href="https://www.ics.uci.edu/~fielding/pubs/dissertation/fielding_dissertation.pdf">Fielding's dissertation</a> describes a Resource as:<br />
<br />
<i>"Any information that can be named" ... "a document or image, a temporal service (e.g. “today’s weather in Los Angeles”), a collection of other resources, a non-virtual object (e.g. a person), and</i><br />
<i>so on. In other words, any concept that might be the target of an author’s hypertext</i><br />
<i>reference must fit within the definition of a resource. A resource is a conceptual mapping</i><br />
<i>to a set of entities, not the entity that corresponds to the mapping at any particular point in</i><br />
<i>time."</i><br />
<br />
Defining a Resource is both a <i>Science </i>and an <i>Art</i>. It requires both Domain knowledge and API Architectural skills. The following points detailed below serve as a checklist which may help you determine the shape of your Resource, what data it should contain and how it should be presented to consumers of your API.<br />
<br />
<h4 style="text-align: left;">
The Resource must contain a Business Description</h4>
<ul style="text-align: left;">
<li>The business description should be 3 - 4 sentences in simple prose which explain what the Resource is. </li>
<li>A developer with a moderate knowledge of your system should be able to understand the description</li>
<li>Any caveats of the Resource should be made clear</li>
</ul>
<h4 style="text-align: left;">
The Resource should be useful on its own</h4>
<br />
This is similar to the maxim of defining the boundary of a micro-service, where a micro-service should be considered to be useful on its own. Similarly, a Resource should be useful on its own.<br />
<br />
For example, instead of:
<br />
<pre class="brush: jscript; ruler: true; first-line: 1">/street-address/{id}
RESPONSE
{
"street1": "String",
"street2": "String"
}
</pre>
and
<br />
<pre class="brush: jscript; ruler: true; first-line: 1">/address-extra/{id}
RESPONSE
{
"city": "String",
"country": "String"
}
</pre>
It should be:
<br />
<pre class="brush: jscript; ruler: true; first-line: 1">/address/{id}
RESPONSE
{
"street1": "String",
"street2": "String",
"city": "String",
"country": "String"
}
</pre>
If a Resource on its own is not useful and <i>always </i>necessitates a subsequent request, it means code will inevitably become more complex as well as there being a performance impact incurred from the second request<br />
<br />
<h4 style="text-align: left;">
Use an Appropriate Noun</h4>
<br />
<div style="text-align: left;">
Use of a simple noun over a compound noun is preferred. For example, <i>Address </i>is better than <i>AddressInfo </i>or <i>AddressDetail</i>. This is a general rule, there will always be <i>exceptions</i>.</div>
<div style="text-align: left;">
<br /></div>
If using multiple Resources to represent different views of the same data, for example: <i>Address </i>and <i>AddressDetail</i>, use the simple noun e.g <i>Address </i>first. Then if the second representation is more detailed use <i>ResourceNameDetail </i>or if it is less detailed use <i>ResourceNameSummary</i>. For example, suppose there is a requirement to introduce an Address type Resource:
<br />
<ol style="text-align: left;">
<li><i>Address </i>is introduced first</li>
<li>If a subsequent view of Address is needed that is more detailed, the new Resource should be called <i>AddressDetail</i></li>
<li>If a subsequent view of Address is needed that is less detailed, the new Resource should be called <i>AddressSummary</i></li>
</ol>
<h4 style="text-align: left;">
If it is only used in a READ does it need to be a Resource?</h4>
<br />
If a Resource is only ever used in a <i>Read</i> request and never a <i>Write </i>(<i>Create, Partial Update, Full Update, Delete, ...</i>) request it is questionable if it needs to be defined as a Resource with its own URI. It could just be added to the parent payload and if there is a concern that payload then becomes too complex, the parent could just provide a sparse query - where the client can decide per API request what it wants returned.<br />
<br />
<h4 style="text-align: left;">
Resources should conform to the uniform interface</h4>
<br />
The uniform interface is a very important part of good API design. It is not just about using special verbs for different requests but also ensuring the data shape is consistent.<br />
<br />
If creates, reads, updates, deletes etc are done in a consistent way, it means code is more consistent, reusable and more maintainable.<br />
<br />
This means:
<br />
<pre class="brush: jscript; ruler: true; first-line: 1">GET /addresses/{id}
</pre>
and
<br />
<pre class="brush: jscript; ruler: true; first-line: 1">GET /addresses
</pre>
must return the same address data structure to represent an Address.
<br />
<pre class="brush: jscript; ruler: true; first-line: 1">GET /addresses/{id}
RESPONSE
{
"id":"546",
"street1": "String",
"street2": "String",
"city": "String",
"country": "String"
}
</pre>
and
<br />
<pre class="brush: jscript; ruler: true; first-line: 1">GET /addresses
RESPONSE
{
"elements": [
{
"id":"546",
"street1": "String",
"street2": "String",
"city": "String",
"country": "String"
},
...
]
}
</pre>
Similarly, for write payloads, the Data Structure should be the same. So, a partial update to change street1 would be:<br />
<br />
<pre class="brush: jscript; ruler: true; first-line: 1">PATCH /addresses/{id}
REQUEST
{
"street1": "Walkview"
}
RESPONSE
{
"id":"546",
"street1": "Walkview",
"street2": "Meadowbrook",
"city": "Dublin",
"country": "Ireland"
}
</pre>
and not something like
<br />
<pre class="brush: jscript; ruler: true; first-line: 1">PATCH /addresses/{id}
REQUEST
{
"newStreet1Value": "Walkview"
}
</pre>
<br />
From a Resource perspective, the data structure <b>must</b> be consistent. A different data structure means a different Resource, which should be named differently and have its own path.<br />
<br />
<h4 style="text-align: left;">
Don't expose everything</h4>
<br />
If your DB model is quite sophisticated, you can be sure not all attributes need to be exposed at an API level. Some fields may only be getting persisted for back office processing and should never presented make it to any UI.<br />
<br />
When adding an attribute to a Resource, consider:<br />
<br />
<ul style="text-align: left;">
<li>to only include fields that you are sure the client is interested in </li>
<li>if you are not sure, leave the attribute out. It is much smaller problem to add an attribute later on, then to remove an attribute that has already been exposed.</li>
</ul>
<br />
<h4 style="text-align: left;">
API Models shouldn't blindly mirror DB Relational model or OO Models</h4>
<br />
In database modelling approaches such as normalizing data or collapsing inheritance hierarchies are used. In Object Orientated design, techniques such as polymorphism, inheritance hierarchies etc are used to promote things like code reuse and to reduce coupling.<br />
<br />
Resource modelling does not have to follow theses techniques. The consumer of an API doesn't care if the data is all in one table, or normalized over multiple tables. In general, the API returns data in a format that is easy to use and does not require much additional mapping by the client before it can become useful.<br />
<br />
<h4 style="text-align: left;">
Use Hierarchical data to Avoid repetition</h4>
<br />
One of the advantages of hierarchical data over flat formats such as CSV is that it provides a mechanism to avoid repetition. For example, consider a flat data structure which contains a list of persons and what team they are in. In CSV this is:<br />
<br />
<pre class="brush: text; ruler: true; first-line: 1">team, firstname, lastname
Liverpool, Mo, Salah
Liverpool, Andy, Roberston
</pre>
<br />
In JSON this could be:<br />
<pre class="brush: jscript; ruler: true; first-line: 1">{
"team": "Liverpool",
"players": [
{
"firstName":"Mo",
"lastName":"Salah"
},
{
"firstName":"Andy",
"lastName":"Roberston"
},
...
]
}
</pre>
<br />
<h4>
Use Hierarchical Data to Make context clear</h4>
<br />
Another advantage of hierarchical data is that it helps provide context. To understand a flat data structure you need to know what the query was that generated the data to understand the meaning of it. For example, consider a bunch
of rows that contain a date range.<br />
<br />
<pre class="brush: text; ruler: true; first-line: 1">name, fromDate, toDate, holidays
Tony, 2018-01-01, 2018-02-02, true
Tony, 2018-02-03, 2018-03-01, false
</pre>
<br />
You could make assumptions that there is a new row when there is a change in Tony being on holidays. But, what if there is another column?<br />
<br />
<pre class="brush: text; ruler: true; first-line: 1">name, fromDate, toDate, holidays, sick
Tony, 2018-01-01, 2018-02-02, true, false
Tony, 2018-02-03, 2018-03-01, false, true
</pre>
<br />
Are the date ranges corresponding to holidays, sickness or both?<br />
<br />
If we get more data back maybe it might be clearer...<br />
<pre class="brush: text; ruler: true; first-line: 1">name, fromDate, toDate, holidays, sick,
Tony, 2018-01-01, 2018-02-02, true, false
Tony, 2018-02-03, 2018-03-01, false, true
Tony, 2018-03-02, 2018-04-01, false, false
</pre>
Now it looks like it's sickness that the date range corresponds to and its only a coincidence it lines up with a holiday period. However, when we get more data back this theory also fails.<br />
<pre class="brush: text; ruler: true; first-line: 1">name, fromDate, toDate, holidays, sick,
Tony, 2018-01-01, 2018-02-02, true, false
Tony, 2018-02-03, 2018-03-01, false, true
Tony, 2018-03-02, 2018-04-01, false, false
Tony, 2018-04-02, 2018-05-01, true, false
</pre>
<br />
It gets even more complicated when just don't have some information. For example:
<br />
<pre class="brush: text; ruler: true; first-line: 1">name, fromDate, toDate, holidays, sick,
Tony, 2018-01-01, 2018-02-02, true, false
Tony, 2018-02-03, 2018-03-01, false, true
Tony, 2018-03-02, 2018-04-01, false, false
Tony, 2018-04-02, 2018-05-01, true, false
Tony, 2018-05-02, 2018-06-01, null, false
Tony, 2018-06-02, 2018-07-01, null, false
Tony, 2018-07-02, 2018-07-08, true, false
Tony, 2018-07-08, 2018-07-09, true, null
</pre>
<br />
The limitation with flat data structures is not only lack of normalisation but that they can only go so far in making the data self-describing.<br />
<br />
When it isn't clear what data means, it is inevitable processing the data will be buggy.<br />
<br />
We could represent the same person data in hierarchical format as:
<br />
<pre class="brush: jscript; ruler: true; first-line: 1">{
"name":"tony",
"holidays": [
{
"fromDate":"2018-01-01",
"toDate":"2018-02-02"
},
{
"fromDate":"2018-04-02",
"toDate":"2018-05-01"
},
{
"fromDate":"2018-07-02",
"toDate":"2018-07-09"
}
],
"sick": [
{
"fromDate":"2018-02-03",
"toDate":"2018-03-01"
}
]
}
</pre>
Now, the data is much more self describing and it is clear when a date range is for a holiday and when it is for a sick period.
<br />
<br />
<h4 style="text-align: left;">
Resource Relationships</h4>
Resources on their own only describe themselves. A Resource model describes relationships between Resources. This will give an indication of:
<br />
<ul>
<li>dependencies between Resources. What Resources are needed
for a particular Resource to exist or what is impacted when a particular Resource changes: updated or deleted.</li>
<li>Data navigation - in a large domain model, it is much easier to understand and follow if navigational and directional sense is provided to consumers of the model. Especially, when to navigation across (Resources loosely connected) can be be differentiated from navigation down (Resources strongly connected)</li>
</ul>
Hypermedia links aren't only used to achieve HATEOAS. Resources that describe what they are linked to using hypermedia links demonstrate a very powerful mechanism to express the Resource model. Advantages include:
<br />
<ul style="text-align: left;">
<li>A large domain model is split into more manageable pieces. Typically users are only interested in a particular part of the model. When Resources self describe their own relationships, it means a large complex model is split up into more digestible chunks and users get the information they need quicker. </li>
<li>The Resource model is self-describing and kept in sync with code. Everything is co-located.</li>
</ul>
<b>Make clear Parent - Child relationships</b><br />
A Child Resource describes its Parent URL hierarchical name spacing. A Parent Resource has children of one or many types should make this clear by providing links to the children. For example, if a Team Resource has Players child Resources. The Team payload should make this clear.<br />
<br />
<pre class="brush: jscript; ruler: true; first-line: 1">REQUEST
https://api.server.com/teams/4676
RESPONSE
{
"id":"34533",
...,
"_links": {
"self":"https://api.server.com/teams/4676",
"players":"https://api.server.com/teams/4676/players"
}
}
</pre>
<h4 style="text-align: left;">
Make clear Peer relationships</h4>
This is similar to above except it is for Resources that exist in a different hierarchical name space. So for example, suppose the team is in division 1. A link should be included in the team's division attribute.<br />
<pre class="brush: jscript; ruler: true; first-line: 1">REQUEST
https://api.server.com/teams/4676
RESPONSE
{
"id":"34533",
"division": {
"name":"Division 1",
"_links": {
"self":"https://api.server.com/divisions/1"
}
},
...,
"_links": {
"self":"https://api.server.com/teams/4676",
"players":"https://api.server.com/teams/4676/players"
}
}
</pre>
<h4 style="text-align: left;">
Make clear Links to Other Representations</h4>
If data is modeled to have multiple Resources which represent different representations of the data, the Resources should also include links
to each other.
<br />
<pre class="brush: jscript; ruler: true; first-line: 1">REQUEST
https://api.server.com/teams/4676
RESPONSE
{
"id":"34533",
"division": {
"name":"Division 1",
"_links": {
"self":"https://api.server.com/divisions/1"
}
},
...,
"_links": {
"self":"https://api.server.com/teams/4676",
"players":"https://api.server.com/teams/4676/players",
"teamDetails":"https://api.server.com/teamDetails/4676"
}
}
</pre>
</div>
Alex Staveleyhttp://www.blogger.com/profile/04378245339254378938noreply@blogger.com1tag:blogger.com,1999:blog-8965734277283465944.post-80854725204203673532018-12-19T14:28:00.002-08:002018-12-20T13:49:05.048-08:00What's the case for your API?<div dir="ltr" style="text-align: left;" trbidi="on">
<i>Disclaimer: In pure REST, API's are opaque and the URL should be whatever what was sent as a link in the response to a previous request. But, I'm not talking pure REST, I'm talking more pragmatic APIs which involve some concepts from REST and as well as general API best practices. </i><br />
<i><br /></i>
When writing an API, it starts simple. You identify the obvious resources and end up with endpoints such as:<br />
<pre class="brush: jscript; ruler: true; first-line: 1">/api.mycompany.com/tweet
</pre>
<br />
Eventually, your API will have to capture more sophisticated concepts and model more complex resources that cannot be expressed in short single nouns. Some real world examples include:<br />
<ul style="text-align: left;">
<li>Enabling request validation via a <i>Request Validator</i> resource (AWS API Gateway API)</li>
<li>Performing a customer search via a <i>Customer Search</i> resource (Google Customer Search API)</li>
<li>Running powerful checks against code via a <i>Check Runs</i> resource (Github API)</li>
</ul>
<div>
In English grammar, nouns that are really <i>two nouns</i> joined in some way are called <i>compound nouns</i> and in English grammar, compound nouns follow one of three patterns:</div>
<div>
<ol style="text-align: left;">
<li>All the one word: haircut, toothpaste</li>
<li>Two words: rain forest, ice cream</li>
<li>Hyphenated: self-esteem, brother-in-law</li>
</ol>
</div>
<div>
In the API world there are different options to choose from but it is better for consistency that your APIs just pick one approach and stick to that it. So firstly, what are the options for compound nouns from an API perspective?<br />
<br /></div>
<h4 style="text-align: left;">
Camel Case</h4>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh3KEIX_zjucl6zt5wPN9O9fIek5dIXxptEpmK3DIysHw38GIOe3w6nTawdjjsPCHyCPoh5f2M0PL9bn57dszi058aziOhEhbfcAZj3zlHYMdgS3iCJt3y2_CmLX4C6IbXk3hlJn0hU63By/s1600/camel.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="249" data-original-width="202" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh3KEIX_zjucl6zt5wPN9O9fIek5dIXxptEpmK3DIysHw38GIOe3w6nTawdjjsPCHyCPoh5f2M0PL9bn57dszi058aziOhEhbfcAZj3zlHYMdgS3iCJt3y2_CmLX4C6IbXk3hlJn0hU63By/s200/camel.jpg" width="162" /></a></div>
<div>
<br /></div>
<div>
<a href="https://en.wikipedia.org/wiki/Camel_case">Camel case</a> is the practise of writing each word in the phrase with a capital letter. There are two variations:<br />
<ol style="text-align: left;">
<li>Initial upper case (also know as <i>Pascal's case</i>) is where the first letter is also a capital, for example: <i>IceCream</i>. Pascal's case is popular in programming languages for naming classes e.g. Java. </li>
<li>Initial lower case is where the initial letter is always lower case, for example: <i>iceCream</i>. This approach is popular in programming languages (<i>again Java is a good example</i>) for naming variables. When people say camel case, they are <i>usually </i>referring to the initial lower case format.</li>
</ol>
</div>
<h4 style="text-align: left;">
</h4>
<h4 style="text-align: left;">
Kebab Case</h4>
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjIDRYLbd5xdiZ3Xri7y6By0dy6XLVoVGPNZNGU5bkDO9UyClohhSdP-FfmiyqO5jrwAid7MOheSp4lEyyZUqJY-YQxpv1JKpbiHoBr28c_tbc7HXQIqxfktNC_RH-tpgZWRHgT8NjXnvEy/s1600/kebab.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="186" data-original-width="271" height="137" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjIDRYLbd5xdiZ3Xri7y6By0dy6XLVoVGPNZNGU5bkDO9UyClohhSdP-FfmiyqO5jrwAid7MOheSp4lEyyZUqJY-YQxpv1JKpbiHoBr28c_tbc7HXQIqxfktNC_RH-tpgZWRHgT8NjXnvEy/s200/kebab.jpg" width="200" /></a></div>
In Kebab Case, the individual words are separated by hyphens. Ice cream is expressed as <i>ice-cream</i>. This approach is used in the <a href="https://en.wikipedia.org/wiki/Naming_convention_(programming)">Lisp programming</a> language, in lots of URLs (for example, every blog post in www.blogger.com e.g. http://dublintech.blogspot.com/2018/08/oauth-20-authorisation-code-grant.html). The observant amongst you will note sometimes the word "<i>dash" </i>is sometimes used in technical references instead of "<i>hyphen".</i> So, what's the difference? In English grammar, hyphen is the thing used to join two words to make one whereas the dash is the thing used to usually add some sort stylistic emphasis to the end of a sentence such as: "I might have an interesting point here <i>- you never know"</i>.</div>
<div>
<br /></div>
<div>
In programming we don't care whether the term is <i>hyphen </i>and <i>dash</i>. They are used interchangeably and mean the same thing.<br />
<br />
The kebab case approach became popular in Web URIs because search engines knew that the hyphen meant separate words and could index the URI properly. This convention used by search engines meant hyphens became a <i>de facto</i> standard for URIs.<br />
<br /></div>
<h4 style="text-align: left;">
</h4>
<h4 style="text-align: left;">
Snake Case</h4>
<div>
<div class="separator" style="clear: both; text-align: center;">
</div>
In this approach, an underscore is used to separate words. Ice cream becomes ice_cream. This approach is used in <a href="https://en.wikipedia.org/wiki/Naming_convention_(programming)">Python and Ruby</a> for anything other than a class name or static constant.<br />
<h4 style="text-align: left;">
</h4>
<h4 style="text-align: left;">
Join words</h4>
</div>
<div>
In this approach the words are just joined. <i>There is no -, no _ and no capitalisation of anything</i>. This is not a popular approach with developers because it's difficult to read.</div>
<h4 style="text-align: left;">
</h4>
<h4 style="text-align: left;">
</h4>
<h4 style="text-align: left;">
APIs</h4>
<div>
Should we use camelCase, kebab-case or snake_case in an API? Well unfortunately, <a href="https://www.ics.uci.edu/~fielding/pubs/dissertation/fielding_dissertation.pdf">Mr. Fielding's dissertation</a> did not go into such detail. So what are people actually doing? And is the approach used consistent across the API's URL and the JSON Body. Let's take a look.</div>
<h4>
</h4>
<h4>
</h4>
<h4>
AWS</h4>
<div>
AWS have different API styles for different services. The <a href="https://docs.aws.amazon.com/apigateway/api-reference/link-relation/requestvalidator-by-id/">API Gateway REST API reference</a> shows that JSON payload uses camel case <i>but </i>the URL uses nothing, it's just:</div>
<div>
<pre class="brush: jscript; ruler: true; first-line: 1">/restapis/{id}/requestvalidators/{requestvalidatorId}</pre>
<h4 style="text-align: left;">
</h4>
<h4 style="text-align: left;">
</h4>
<h4 style="text-align: left;">
Google</h4>
<div>
Surprise, surprise Google also have <a href="https://developers.google.com/apis-explorer/#p/">lots of APIs</a>. The Google<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhQ3tTBrYGNJMAlCDKIz6xamxsbPYEdxu42369fQsSRUN1HYNjQNz2P8P4utJ3OJesh94zRLhAaGPtimDyLos3raiSN5CCfgDBJIUBBa0xALZQ3HewoN03q-drbJgSjQh8mTlUgHOmYn-_N/s1600/google-apis.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="388" data-original-width="420" height="184" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhQ3tTBrYGNJMAlCDKIz6xamxsbPYEdxu42369fQsSRUN1HYNjQNz2P8P4utJ3OJesh94zRLhAaGPtimDyLos3raiSN5CCfgDBJIUBBa0xALZQ3HewoN03q-drbJgSjQh8mTlUgHOmYn-_N/s200/google-apis.jpg" width="200" /></a></div>
<a href="https://developers.google.com/custom-search/v1/cse/list">Custom Search API</a> is similar to the AWS API Gateway API. The compound noun in the URL is just the one word and the JSON body is camel case. </div>
<div>
<br /></div>
<div>
The Google Gmail API has camel case in request body and in some URLs, for example the <a href="https://developers.google.com/gmail/api/v1/reference/users/settings/forwardingAddresses/create">forwarding addresses API</a>. </div>
<div>
<br /></div>
<div>
The Google <a href="https://developers.google.com/youtube/analytics/reference/reports/query">youtube API</a> sometimes will use kebab case in the URL e.g. <i>yt-analytics</i> but in other cases will use single word e.g. <i>youtubepartner. </i> But the JSON payload is camel case.</div>
<h4 style="text-align: left;">
</h4>
<h4 style="text-align: left;">
</h4>
<h4 style="text-align: left;">
Github</h4>
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgZ7M27RTYFYqEOpjuNQrjMbwZ8KPpG5UrEL7Rl1wGHu9xAKr700P-rPUboD-voQRJM8U6xoD7pIVelJ1pY79MOJI0VhTQDudx2fRWpb_Q59YA3b7b7Yol4Gm2bWLPjBeFbPKNGQDUmAejB/s1600/github-logo.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="1200" data-original-width="1200" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgZ7M27RTYFYqEOpjuNQrjMbwZ8KPpG5UrEL7Rl1wGHu9xAKr700P-rPUboD-voQRJM8U6xoD7pIVelJ1pY79MOJI0VhTQDudx2fRWpb_Q59YA3b7b7Yol4Gm2bWLPjBeFbPKNGQDUmAejB/s200/github-logo.png" width="200" /></a></div>
The Github API is a good example where we get a reminder that if possible, you should try to avoid this issue by trying to avoid compound nouns as it avoids them by using some creative name spacing.</div>
<div>
<br />
However, some more rooting around and you'll find a compound noun such as check run expressed using kebab case in the URL and he JSON body using snake case. </div>
<h4 style="text-align: left;">
</h4>
<h4 style="text-align: left;">
</h4>
<h4 style="text-align: left;">
Stripe</h4>
<div>
Stripe use snake case in the URL and in the JSON body. For example the <a href="https://stripe.com/docs/api/payment_intents/create">PaymentsIntents API</a>. </div>
<div>
<br /></div>
<div>
<pre class=" language-bash" style="background: rgb(39, 43, 45); border-radius: 5px; border: 0px; color: #d0d0d0; direction: ltr; font-size: 13px; hyphens: none; line-height: 1.5em; outline: 0px; padding: 20px 40px; tab-size: 4; vertical-align: baseline; white-space: pre-wrap; word-break: break-word;"><code class=" language-bash" style="border: 0px; direction: ltr; font-style: inherit; font-weight: inherit; hyphens: none; margin: 0px; outline: 0px; padding: 0px; tab-size: 4; vertical-align: baseline; word-break: break-word; word-spacing: normal;"> https://api.stripe.com/v1/payment_intents </code></pre>
</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
and JSON body...</div>
<pre class=" language-json" style="background: rgb(39, 43, 45); border-radius: 5px; border: 0px; color: #d0d0d0; direction: ltr; font-size: 13px; hyphens: none; line-height: 1.5em; outline: 0px; padding: 20px 40px; tab-size: 4; vertical-align: baseline; white-space: pre-wrap; word-break: break-word;"><code class=" language-json" style="border: 0px; direction: ltr; font-style: inherit; font-weight: inherit; hyphens: none; margin: 0px; outline: 0px; padding: 0px; tab-size: 4; vertical-align: baseline; word-break: break-word; word-spacing: normal;"><span class="token punctuation" style="border: 0px; color: #d0d4d7; font-family: inherit; font-style: inherit; font-weight: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">{</span><span class="token json-key" style="border: 0px; color: #c1ef65; font-family: inherit; font-style: inherit; font-weight: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
"id"</span><span class="token punctuation" style="border: 0px; color: #d0d4d7; font-family: inherit; font-style: inherit; font-weight: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">:</span> <span class="token json-string" style="border: 0px; font-family: inherit; font-style: inherit; font-weight: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">"pi_Aabcxyz01aDfoo"</span><span class="token punctuation" style="border: 0px; color: #d0d4d7; font-family: inherit; font-style: inherit; font-weight: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">,</span><span class="token json-key" style="border: 0px; color: #c1ef65; font-family: inherit; font-style: inherit; font-weight: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
"object"</span><span class="token punctuation" style="border: 0px; color: #d0d4d7; font-family: inherit; font-style: inherit; font-weight: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">:</span> <span class="token json-string" style="border: 0px; font-family: inherit; font-style: inherit; font-weight: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">"payment_intent"</span><span class="token punctuation" style="border: 0px; color: #d0d4d7; font-family: inherit; font-style: inherit; font-weight: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">,</span><span class="token json-key" style="border: 0px; color: #c1ef65; font-family: inherit; font-style: inherit; font-weight: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
"allowed_source_types"</span><span class="token punctuation" style="border: 0px; color: #d0d4d7; font-family: inherit; font-style: inherit; font-weight: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">:</span> <span class="token punctuation" style="border: 0px; color: #d0d4d7; font-family: inherit; font-style: inherit; font-weight: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">[</span><span class="token json-key" style="border: 0px; color: #c1ef65; font-family: inherit; font-style: inherit; font-weight: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
"card"</span>
<span class="token punctuation" style="border: 0px; color: #d0d4d7; font-family: inherit; font-style: inherit; font-weight: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">]</span><span class="token punctuation" style="border: 0px; color: #d0d4d7; font-family: inherit; font-style: inherit; font-weight: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">,</span><span class="token json-key" style="border: 0px; color: #c1ef65; font-family: inherit; font-style: inherit; font-weight: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
"amount"</span><span class="token punctuation" style="border: 0px; color: #d0d4d7; font-family: inherit; font-style: inherit; font-weight: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">:</span> <span class="token number" style="border: 0px; color: #77bcd7; font-family: inherit; font-style: inherit; font-weight: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">1099</span><span class="token punctuation" style="border: 0px; color: #d0d4d7; font-family: inherit; font-style: inherit; font-weight: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">,</span><span class="token json-key" style="border: 0px; color: #c1ef65; font-family: inherit; font-style: inherit; font-weight: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
"amount_capturable"</span><span class="token punctuation" style="border: 0px; color: #d0d4d7; font-family: inherit; font-style: inherit; font-weight: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">:</span> <span class="token number" style="border: 0px; color: #77bcd7; font-family: inherit; font-style: inherit; font-weight: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">1000</span><span class="token punctuation" style="border: 0px; color: #d0d4d7; font-family: inherit; font-style: inherit; font-weight: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">,</span></code></pre>
<h4 style="text-align: left;">
</h4>
<h4 style="text-align: left;">
</h4>
<h4 style="text-align: left;">
Paypal</h4>
<div>
Paypal have more compound nouns than the other APIs checked. APIs for resources such as <a href="https://developer.paypal.com/docs/api/overview/#api-requests">billing agreement</a> the API will use kebab case in the URL but then use snake case in the JSON payloads.</div>
<h4 style="text-align: left;">
</h4>
<h4 style="text-align: left;">
</h4>
<h4 style="text-align: left;">
Twitter</h4>
<div>
<a href="https://developer.twitter.com/en/docs/api-reference-index">Twitter</a> use snake case in the URL e.g. /saved_searches/ and snake case in the JSON payloads.</div>
<h4 style="text-align: left;">
</h4>
<h4 style="text-align: left;">
</h4>
<h4 style="text-align: left;">
Facebook</h4>
<a href="https://developers.facebook.com/docs/graph-api">Facebook's Graph API</a> tends to avoid resource naming in URLs and in JSON bodies it is snake case.<br />
<br />
By this stage, you should be getting a little but confused. So let's recap via the table below.<br />
<br />
<table style="width: 100%;">
<tbody>
<tr>
<th>API</th><th>URL</th><th>JSON Body</th><th><br /></th>
</tr>
<tr>
<td><a href="https://docs.aws.amazon.com/apigateway/api-reference/index.html">AWS API Gateway</a></td><td>No separator</td><td>camelCase</td></tr>
<tr>
<td><a href="https://developers.facebook.com/docs/graph-api/reference/live-video">Facebook Graph API</a> </td><td>N/A</td><td>snake_case</td>
</tr>
<tr>
<td><a href="https://developer.github.com/v3/checks/runs/">Github</a>
</td><td>Snake and Kebab</td><td>snake_case</td>
</tr>
<tr>
<td><a href="https://developers.google.com/custom-search/v1/cse/list">Google custom search</a>
</td><td>No separator</td><td>camelCase</td>
</tr>
<tr>
<td><a href="https://developers.google.com/gmail/api/v1/reference/users/settings/forwardingAddresses/get">Google Gmail</a>
</td><td>camelCase</td><td>camelCase</td>
</tr>
<tr>
<td><a href="https://developer.linkedin.com/docs/ref/v2/standardized-data/iab-categories">LinkedIn</a>
</td><td>camelCase</td><td>camelCase</td>
</tr>
<tr>
<td><a href="https://developer.paypal.com/docs/api/payments.billing-agreements/v1/">Pay pal</a>
</td><td>kebab-case</td><td>snake_case</td>
</tr>
<tr>
<td><a href="https://stripe.com/docs/api/payment_intents/create">Stripe</a>
</td><td>snake_case</td><td>snake_case</td>
</tr>
<tr>
<td><a href="https://developer.twitter.com/en/docs/accounts-and-users/manage-account-settings/api-reference/get-saved_searches-list">Twitter</a>
</td><td>snake_case</td><td>snake_case</td>
</tr>
</tbody></table>
<h4 style="text-align: left;">
</h4>
<h4 style="text-align: left;">
</h4>
<h4 style="text-align: left;">
</h4>
<h4 style="text-align: left;">
<br /></h4>
<h4 style="text-align: left;">
Everyone is different, what should I do?</h4>
<div>
So there is a lack of consistency across the industry. However there are point worth making:<br />
<ol style="text-align: left;">
<li>In general compound nouns are best avoided. In all the APIs checked (except PayPal), they appear in under 5% of the APIs. <i>This means developers don't get upset when their favourite approach is not used.</i></li>
<li>The only Web API in the selection above that had more than 5% of its APIs using compound nouns was PayPal and they went for kebab-case in URIs.</li>
<li>kebab-case is never used in any JSON body. The syntax is allowed. So what drives this trend? It's more than likely because JavaScript Web UIs are possibly the mos popular client invoking API and the similarly the most popular back end language serving the API is Java and both of those dudes don't allow the - in any of their declarations.</li>
</ol>
<h4 style="text-align: left;">
</h4>
<h4 style="text-align: left;">
Is there anyone else saying anything?</h4>
<br />
<div>
<div>
In the excellent <a href="http://shop.oreilly.com/product/0636920021575.do">REST API Design Cookbook</a>, industry expert <a href="https://www.linkedin.com/in/markmasse/">Mark Masse</a> suggests:</div>
<div>
<ol style="text-align: left;">
<li>Make your APIs lower case, which rules out camel case</li>
<li>Use kebab case when expressing compound terms in your URIs</li>
<li>Avoid using underscores as they may not display well since some browser render hyperlinks with an underline added</li>
</ol>
</div>
</div>
<h4 style="text-align: left;">
</h4>
<h4 style="text-align: left;">
Make a decision</h4>
<div>
<ol style="text-align: left;">
<li>Avoid compound nouns if you can. <i>This isn't always possible.</i> Sticking to a <a href="https://martinfowler.com/bliki/UbiquitousLanguage.html">ubiquitous language</a> is important and helpful. If you have a complex business application you will have lots of compound nouns. </li>
<li>If you can't avoid compound nounds and more than 5% of the APIs are going to involve compound nouns use kebab case for your URIs. Why? Because if you have a complex business domain it's not only developers you need to think about. Lots of BA's, Product Architects, curious Managers will also be looking at your APIs. Kebab-case is the easiest to read for everyone.</li>
<li>For JSON body, I think it is okay to use camelCase because this is the easiest to map back to JavaScript and Java code. It is also a recommendation from Google to use <a href="https://google.github.io/styleguide/jsoncstyleguide.xml">camelCase</a> in JSON.</li>
<li>If you have to use camelCase in your URI's, consider using the first letter capital approach for the URIs as the URIs are supposed to marking resources not attributes. Resources are more analogous to Java Classes which also use initial letter capital format; whereas the JSON payload attributes are analogous to Java attributes which use initial lower case.</li>
</ol>
</div>
Until the next time, take care of yourselves.<br />
<br /></div>
<br />
<br /></div>
</div>
Alex Staveleyhttp://www.blogger.com/profile/04378245339254378938noreply@blogger.com2tag:blogger.com,1999:blog-8965734277283465944.post-90729229130856545472018-11-19T14:54:00.002-08:002018-11-30T15:20:40.426-08:00Technical Debt will kill your Agile Dreams<div dir="ltr" style="text-align: left;" trbidi="on">
<div pid="1" style="background-color: white; box-sizing: border-box; color: #222635; font-family: Cambria, serif; font-size: 19px; margin-bottom: 15px; margin-top: 5px; padding-top: 0px;">
<em style="box-sizing: border-box;">Warning this blog post is a bit of a rant, if you want something more technical there's plenty of other articles on this blog and many others!</em></div>
<div pid="2" style="background-color: white; box-sizing: border-box; color: #222635; font-family: Cambria, serif; font-size: 19px; margin-bottom: 15px; margin-top: 5px; padding-top: 0px;">
<a href="https://en.wikipedia.org/wiki/Technical_debt" rel="nofollow" style="background: transparent; box-sizing: border-box; color: #29a8ff; text-decoration-line: none;" target="_blank">Technical debt</a> is a concept used in software engineering to express the additional complexity that is added to a project due to technical decisions that result in inferior solutions being chosen because they can be delivered quicker. It is analogous to <em style="box-sizing: border-box;">financial debt</em>. Borrow money for some gain, incur debt but -<em style="box-sizing: border-box;">and there's always abut -</em> you have to pay the debt off. Otherwise, with interest, the debt will grow and you will eventually be bankrupt. Similarly, it is ok to incur some technical debt. If you spend too long looking for the perfect solution your customers will have moved over to someone else. At the same time, if you incur too much technical debt and you are not meeting your repayments, just as the interest on the financial debt will compound repayments, if you don't meet your technical debt repyaments the complexity and entropy increases with time and eventually it will stagnate your product. Something that should really only take one month to deliver, suddenly takes 3 months, then 6 months, then there's only a few people in the entire company who can do it in 6 months, then a few of them leave, then...</div>
<div pid="3" style="background-color: white; box-sizing: border-box; color: #222635; font-family: Cambria, serif; font-size: 19px; margin-bottom: 15px; margin-top: 5px; padding-top: 0px;">
Where the analogy breaks down is that all too often technical debt is also used to express that there has just been some bad engineering. Bad engineering decisions are in a different category to ones that were tactically made with full knowledge that the short-term priority was worth it. When it's clear that such a decision was, in fact, a tactical decision, it is much easier to convince people that refactoring needs to happen and the debt has to be paid off. Unfortunately, when the term is used as a polite way of saying bad engineering, it's unlikely there is any repayment strategy in place and it is even harder to create one because first, you need to convince people there is some bad engineering, then you need to convince people it is causing problems, then you have to have the ability to think of a better approach, cost it and then convince the various stakeholders the investment is worth it. It is like trying to win 5 matches in a row away from home when the odds are against you. </div>
<div pid="5" style="background-color: white; box-sizing: border-box; color: #222635; font-family: Cambria, serif; font-size: 19px; margin-bottom: 15px; margin-top: 5px; padding-top: 0px;">
So, what are the signs that your technical debt (irrespective of whether it is intentional or accidental) is just too high and you could be filing for bankruptcy soon?</div>
<div pid="6" style="background-color: white; box-sizing: border-box; color: #222635; font-family: Cambria, serif; font-size: 19px; margin-bottom: 15px; margin-top: 5px; padding-top: 0px;">
Well, think about this. In the Agile world, we want to develop, build, release in quick cycles and get rapid feedback from the customer and go again. This is only possible if there is an abundance of high quality and well engineered automated tests that run fast and provide confidence that a change no matter what it is has not broken anything. It doesn't matter if the breakdown of the tests are: 68% unit tests and 32% are integration tests or 91% are unit tests and 9% integration tests, the tests have to run fast and they have to provide confidence to all stakeholders. Otherwise releasing will be a pain and it will not be possible to release regularly. That means being Agile and getting all the benefits of it will be very difficult — no matter how good your backlog grooming sessions are.</div>
<div pid="7" style="background-color: white; box-sizing: border-box; color: #222635; font-family: Cambria, serif; font-size: 19px; margin-bottom: 15px; margin-top: 5px; padding-top: 0px;">
What usually makes it difficult for developers to write good tests? Well, it's usually technical debt. It doesn't matter if it is intentional or accidental.</div>
<div pid="8" style="background-color: white; box-sizing: border-box; color: #222635; font-family: Cambria, serif; font-size: 19px; margin-bottom: 15px; margin-top: 5px; padding-top: 0px;">
Now, there are all sorts of tools that will measure technical debt and put it up on a nice looking <a href="https://www.sonarqube.org/" rel="nofollow" style="background: transparent; box-sizing: border-box; color: #29a8ff; text-decoration-line: none;" target="_blank">SonarQube</a> board but usually these tools only pick up the trivial stuff — removing an unused import etc. Who cares? Such trivial stuff isn't going to slow anyone down. The real technical debt problems are the ones that slow people down, they make it harder to make changes, fix bugs, add functionality and do it all quickly with confidence - why because they have made things much harder to test. Sadly, these are technical problems are usually not just something an IDE, a PMD or Checkstyle will prompt you to do. They are generally much deeper in nature - towards the architectural end of specturm. Some examples:</div>
<ul style="background-color: white; box-sizing: border-box; color: #222635; font-family: Cambria, serif; font-size: 19px; margin-bottom: 10px; margin-top: 0px;">
<li style="box-sizing: border-box; padding: 0px 0px 8px;">Lack of data encapsulation and immutability leading to huge cyclomatic complexity, unpredictable code paths and difficulty in predicting impacts.</li>
<li style="box-sizing: border-box; padding: 0px 0px 8px;">Impedence mismatches</li>
<li style="box-sizing: border-box; padding: 0px 0px 8px;">Lack of modularity and too much coupling</li>
<li style="box-sizing: border-box; padding: 0px 0px 8px;"><em style="box-sizing: border-box;">Lack of </em>or <em style="box-sizing: border-box;">bad application</em> of patterns</li>
<li style="box-sizing: border-box; padding: 0px 0px 8px;">A proprietary language introduced with no IDE support, no mechanism to easily unit test or debug code. Zero support from Stackoverflow.</li>
<li style="box-sizing: border-box; padding: 0px 0px 8px;">Spawning of threads and asynchronous call paths when are better approaches which would be much easier to test</li>
</ul>
<div pid="9" style="background-color: white; box-sizing: border-box; color: #222635; font-family: Cambria, serif; font-size: 19px; margin-bottom: 15px; margin-top: 5px; padding-top: 0px;">
This is where the analogy of technical debt and financial debt breaks down. When you have architectural debt you have<em style="box-sizing: border-box;"> big problems</em>. When you have a HashMap used when an ArrayList would have made more sense, you don't. Financial debt doesn't have such a critical distinction. Unless we say, the debt is due to a friendly and sympathetic bank you have a good relationship with or its debt due to some lunatic load shark who will call around to your house with a baseball bat. </div>
<div pid="10" style="background-color: white; box-sizing: border-box; color: #222635; font-family: Cambria, serif; font-size: 19px; margin-bottom: 15px; margin-top: 5px; padding-top: 0px;">
So, if you realize you are approaching your credit limit, what do you do? Firstly, you need to get confidence in your tests. They need to test key functionality and they need to be maintainable. That is more important than speed. If you can't get confidence, you can't ship. There is not much use with tests that run in 5 minutes if no-one has any confidence the functionality you deliver will actually work. Secondly, once you have confidence in the tests (even if they are ugly end-to-end tests), you need to get them to run fast. This is where you can start refactoring — the end to end tests should facilitate refactoring of parts of the call paths; there may be obvious <a href="https://creativeknowledgepool.wordpress.com/2017/10/22/refactoring-to-put-code-under-test-sprout-class-round-1/" rel="nofollow" style="background: transparent; box-sizing: border-box; color: #29a8ff; text-decoration-line: none;" target="_blank">sprouts</a> for example and this should help to move towards a classical <a href="https://martinfowler.com/bliki/TestPyramid.html" rel="nofollow" style="background: transparent; box-sizing: border-box; color: #29a8ff; text-decoration-line: none;" target="_blank">test pyramid</a>. </div>
<div pid="11" style="background-color: white; box-sizing: border-box; color: #222635; font-family: Cambria, serif; font-size: 19px; margin-bottom: 15px; margin-top: 5px; padding-top: 0px;">
Thirdly, you now need to understand why the code is so difficult to achieve high-quality tests. Too much coupling, bad exception handling, bad decompositions it's probably a long list. Understanding why your code is difficult to test is a key architectural and engineering skill as it requires the ability to not just understand the complexity but the ability to be able to know how to reduce it. Reducing the debt should then provide a pathway to make the code easier to test and thus make good tests run fast. Achieving this means you are winning the battle against eventual project stagnation. </div>
<div pid="12" style="background-color: white; box-sizing: border-box; color: #222635; font-family: Cambria, serif; font-size: 19px; margin-bottom: 15px; margin-top: 5px; padding-top: 0px;">
So, the last part of the rant. There is a growing problem with the application of Agile which means we end very easily end up with <a href="https://martinfowler.com/articles/agile-aus-2018.html" rel="nofollow" style="background: transparent; box-sizing: border-box; color: #29a8ff; text-decoration-line: none;" target="_blank">faux Agile</a>. The various Agile books, courses, blogs will detail things like story points, burn downs, stand ups. That stuff is all good but there is not enough focus on technical excellence. Without the technical excellence you get inevitable architectural problems and code that is difficult to test irrespective of your best efforts at story pointing, backlog grooming, sticking yellow post-its up on walls, having arguments over the definition of done, doing your show and tells. All that stuff is nice and beneficial but in comparison to technical excellence it is almost superficial. It is something people can easily see but on its own it never captures the complexity of technical debt. It is the ratio of technical debt to technical excellence that determines whether you can write testable code easily and thus be able to deliver in regular short iteration which is the goal of Agile. Isn't it? </div>
<div pid="13" style="background-color: white; box-sizing: border-box; color: #222635; font-family: Cambria, serif; font-size: 19px; margin-bottom: 15px; margin-top: 5px; padding-top: 0px;">
Lastly, something we all say to our kids when they try any sport: "If you are losing kido, never give up, play to the end of the match". </div>
<div pid="13" style="background-color: white; box-sizing: border-box; color: #222635; font-family: Cambria, serif; font-size: 19px; margin-bottom: 15px; margin-top: 5px; padding-top: 0px;">
Technical Debt is natural, it happens in every project in the world. Your job is to keep it in check and when it gets too high to be able innovate your way work around it and eventually lesson it. That's such an essential characteristic of a good Architect, it is also a handy interview question. </div>
<div pid="14" style="background-color: white; box-sizing: border-box; color: #222635; font-family: Cambria, serif; font-size: 19px; margin-bottom: 15px; margin-top: 5px; padding-top: 0px;">
"Mr. Candidate, describe some Technical Debt you have experienced, describe the effects of it and what strategies you put in place to either work around it or deal with it?"</div>
<div pid="16" style="background-color: white; box-sizing: border-box; color: #222635; font-family: Cambria, serif; font-size: 19px; margin-bottom: 15px; margin-top: 5px; padding-top: 0px;">
Or even, </div>
<div pid="15" style="background-color: white; box-sizing: border-box; color: #222635; font-family: Cambria, serif; font-size: 19px; margin-bottom: 15px; margin-top: 5px; padding-top: 0px;">
"Mr. Candidate here are some examples of Technical Debt, how would you prioritise which ones to deal with? Could you describe some better solutions and how would you convince people they are worth it?"</div>
<div pid="14" style="background-color: white; box-sizing: border-box; color: #222635; font-family: Cambria, serif; font-size: 19px; margin-bottom: 15px; margin-top: 5px; padding-top: 0px;">
The quality of detail in the answer you get back is very likely to indicate the level of pragmatism, experience and the innovation skills of the Architect / Engineer.</div>
<span style="background-color: white; color: #222635; font-family: Cambria, serif; font-size: 19px;">Until the next take care of yourselves.</span></div>
Alex Staveleyhttp://www.blogger.com/profile/04378245339254378938noreply@blogger.com0tag:blogger.com,1999:blog-8965734277283465944.post-70638793744619125822018-08-05T13:43:00.005-07:002018-08-05T14:02:46.241-07:00OAUTH 2.0 Authorisation Code Grant<div dir="ltr" style="text-align: left;" trbidi="on">
<a href="https://tools.ietf.org/html/rfc6749">OAuth 2.0</a> provides a number of security flows (or grant types) to allow an application access user's data in another application. In this blog, we will look at the<br />
OAuth 2.0 grant: Authorisation Code Grant.<br />
<br />
Firstly, a number of definitions:<br />
<ul style="text-align: left;">
<li><b>Client</b>: The application the user is currently interacting with. For example, let's assume a fictitious funky blogging site: www.myfunkyblog.com. The client wants to communicate with another application and retrieve something about the user from there. For example, their favourite photo! Let's assume the fictitious <i>megaphotosharing.com </i>as the service<i> </i>the client wishes to access. </li>
<li><b>Client ID</b>: this is an ID which identifies the client. It can be passed around openly in Web URLs etc</li>
<li><b>Client secret ID</b>: A secret ID that <b>only</b> the client knows. This is kept server side and will be used in requests to the application for which access is sought. It <b><i>cannot</i></b> be passed around in Web URLs. </li>
<li><b>Resource owner</b>: This is usually <i>the human</i>, who is using the client application. The resource owner has data in another application (e.g. megaphotosharing.com) that the client (<i>myfunkyblog.com</i>) wishes to access. The goal is to facilitate that sharing without the need for the Resource owner aka the human to ever pass their <i class="">megaphotosharing.com</i> password to <i>myfunkyblog.com</i>. Note: the resource owner doesn't have to be a human but interestingly according to the <a href="https://tools.ietf.org/html/rfc6749">OAuth spec</a>, when it is a human it can also be referred to as the end - user. </li>
<li><b>Resource Server</b>: hosts the protected resources of the resource owner that the client is interested in. So this is the <i class="">megaphotosharing.com </i>server that has the Resource owner photos the <i>myfunkyblog.com</i> is interested in.</li>
<li><b>Authorisation Server</b>: the server who issues a token to <i>myfunkyblog.com</i> after the resource owner has successfully authenticated and allowed <i>myfunkyblog.com </i>obtain some of its <i class="">megaphotosharing.com</i>. Sometimes the Authorisation Server and the Resource server are effectively the same but they don't have to be. </li>
<li><b>Access token</b>: a special type of token the <i>myfunkyblog.com </i>authorisation server gives <i class="">megaphotosharing.com</i> to access the protected resources. It will contain scope, lifetime and other access attributes.</li>
</ul>
<h4 style="text-align: left;">
Use case</h4>
So the use case is that the client (<i>myfunkyblog.com</i>) wants to access information about<br />
the resource owner (the human) from another application: <i class="">megaphotosharing.com.</i><br />
<h4 style="text-align: left;">
Client Registration</h4>
The first the client must do is register with the service (<i class="">megaphotosharing.com</i>)<br />
providing its name, website etc. The service will return a secret client code.<br />
The client keeps this private and is responsible for ensuring only it knows it. Usually,<br />
it will encrypt and persist it in the client somewhere on the backend. The service will<br />
also receive a client id. Unlike the client secret, this is public and can be passed around<br />
in URLs etc.<br />
<h4 style="text-align: left;">
Flow</h4>
Ok so now the actual flow. The user is browsing around myfunkyblog.com and accesses a part of the site where myfunkyblog.com wants to know what the end-user's favourite photo is.<br />
<br />
A pop-up screen appears to end-user.<br />
<br />
This has URL:<br />
<br />
<pre class="brush: text; ruler: true; first-line: 1">https://megaphotosharing.com/oauth/authorize?response_type=code&client_id=CLIENT_ID&redirect_uri=CALLBACK_URL&scope=read</pre>
<br />
Key parts of this URL:<br />
<ul>
<li><b>megaphotosharing.com:</b> This is the domain for the authorisation server<br />
</li>
<li><b>response_type=code:</b> Required parameter to enable the client informs the authorization server the desired grant type. An alternative value would be the "token", this is for the implicit flow.<br />"code" means the client wants an <i>authorization code</i> which will be returned after<br />resource owner logs in. This authorization code will be used in a subsequent request by the Client. </li>
<li><b>client_id:</b> Required parameter, to identify the client. Remember this is public and<br />can be passed to and from a web browser.<br />
</li>
<li><b>redirect_uri: </b>This is an optional parameter. It enables the client to dynamically specify the URL the auth server should redirect to. <i> In some flows, this isn't needed as there is only one redirect URI and this is registered by the client with the service during client registration.</i><br />
</li>
<li><b>scope: </b> This is an optional parameter. It specifies the level of access that the application is requesting. In this case it is just a read. The auth server uses this to inform the user / resource owner what the client is trying to do.</li>
</ul>
The user then logs into megaphotosharing.com who tells the user what the Client wants to do. If the user selects ok, megaphotosharing.com redirects to the redirect URI passed up.<br />
<br />
<pre class="brush: text; ruler: true; first-line: 1">https://myfunkyblog.com/callback?code=212132kjhkhj</pre>
<br />
Notice how the client id is passed <i>over the web </i>in a URL<i> </i>and the authorisation code is passed back <i>over the web.</i><br />
<br />
The client, then uses the returned authorization code, its client id, client secret and the grant type to make a POST request <b>Server to Server</b> to get an Access Token. This happens all on the back end.<br />
<br />
<pre class="brush: text; ruler: true; first-line: 1">https://megaphotosharing.com/v1/oauth/token?client_id=CLIENT_ID&client_secret=CLIENT_SECRET&grant_type=authorization_code&code=<span style="font-family: "times"; white-space: normal;"> 212132kjhkhj</span>&redirect_uri=CALLBACK_URL</pre>
<br />
Notes:<br />
<br />
<ul style="text-align: left;">
<li><b>client id </b>and <b>client secret</b> identify the client. This is a back-end request and hence it is okay to pass the client_secret (which would obviously never pass to or from the browser).</li>
<li><b>grant_type: </b>This must be set to authorisation_code. As it indicates the Authorisation Code Grant. Remember, the grant is used to indicate the flow the client is using (it can also be used by the server what types of flows are available). <i>If the client was using the Client Credentials Grant, this value would be: "client_credentials". If the client was using "Resource Owner Password Credentials Grant" the value would be "password".</i></li>
<li><b>code: </b>212132kjhkhj - The actual authorisation code what was returned from initial authorisation request from the authorisation server. This is required.</li>
<li><b>redirect_uri: </b>if the redirect_uri was included in the authorisation request this value must be the same as the value used in that request.</li>
</ul>
<br />
<br />
The client then receives back an access token. Something like this:<br />
<br />
<pre class="brush: jscript; ruler: true; first-line: 1">{"access_token":"ACCESS_TOKEN","token_type":"bearer","expires_in":2592000,"refresh_token":"REFRESH_TOKEN","scope":"read","uid":1001013121222}</pre>
<br />
It will now use this to access some of the resource owner's resource data.<br />
<h4 style="text-align: left;">
So what's the big deal?</h4>
<div>
<ul style="text-align: left;">
<li>There are obviously big advantages for users not having to tell one website its password for another site. </li>
<li>Reduces the number of passwords a user needs to remember</li>
<li>Allows richer websites by allowing disparate applications to talk to each other. </li>
</ul>
</div>
<h4 style="text-align: left;">
Why do people find it confusing?</h4>
<div>
There are a number of reasons why people find OAuth 2.0 confusing. </div>
<div>
<ul style="text-align: left;">
<li>There are a few different flows or grants. The <i>Authorisation Code Grant</i> is just one. Sometimes when you google explanations for OAuth 2.0 you get explanations for different grants without making it clear what is and isn't being explained. Hence why I put Authorisation Code Grant in the title. </li>
<li>Terminology. I'll just speak for myself. But if I am reading quickly, I am likely to:</li>
<ul>
<li> confuse "Client" with the end-user</li>
<li>get confused between the Resource Server and Authorisation Server</li>
</ul>
<li>Consistenty. A lot of places implement OAuth 2.0 or something very similar to OAuth but will refer to things differently along the way. For example, go to quora.com and try to login to google. You are taken to:
<pre class="brush: jscript; ruler: true; first-line: 1">
https://accounts.google.com/signin/oauth/oauthchooseaccount?client_id=917071888555.apps.googleusercontent.com&as=rdWeinbqWJbt6ChoW2f3Fg&destination=https%3A%2F%2Fwww.quora.com&approval_state=!ChRyQlhnbEYzai1xQTliNlNmTEVmNRIfZ3doM2hlRVIycGdiMEVBN1JaNXdOM085MERXLVVCWQ%E2%88%99ANKMe1QAAAAAW2i2to0SOyO2_w3k3O4gjwUKQLGNmZ2h&oauthgdpr=1&xsrfsig=AHgIfE8EzSxvWfzyxou0dwLDxv4GhD6e5g&flowName=GeneralOAuthFlow
</pre>
There's no response_type in that URL.
</li>
<li>OAuth is an authorisation spec. It is usually used with Authentication spec like Open Connect but that is actually a separate spec. </li>
</ul>
</div>
<div>
<br /></div>
</div>
Alex Staveleyhttp://www.blogger.com/profile/04378245339254378938noreply@blogger.com1tag:blogger.com,1999:blog-8965734277283465944.post-867245479797921222018-07-27T14:05:00.000-07:002018-08-06T05:34:56.123-07:00Java Lambda Streams and Groovy Clouses Comparisons<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="text-align: left;" trbidi="on">
This Blog post will look at some proverbial operations on List data structure and make some comparison between Java 8/9 and Groovy syntax. So firstly, the data structure. It's just a simple Rugby player who has name and a rating.<br />
<br />
<h4 style="text-align: left;">
Java</h4>
<pre class="brush: java; ruler: true; first-line: 1">class RugbyPlayer {
private String name;
private Integer rating;
RugbyPlayer(String name, Integer rating) {
this.name = name;
this.rating = rating;
}
public String toString() {
return name + "," + rating;
}
public String getName() {
return name;
}
public Integer getRating() {
return rating;
}
}
//...
//...
List<RugbyPlayer> players = Arrays.asList(
new RugbyPlayer("Tadgh Furlong", 9),
new RugbyPlayer("Bundee AKi", 7),
new RugbyPlayer("Rory Best", 8),
new RugbyPlayer("Jacob StockDale", 8)
);
</pre>
<br />
<h4 style="text-align: left;">
Groovy</h4>
<pre class="brush: groovy; ruler: true; first-line: 1">@ToString
class RugbyPlayer {
String name
Integer rating
}
//...
//...
List<RugbyPlayer> players = [
new RugbyPlayer(name: "Tadgh Furlong", rating: 9),
new RugbyPlayer(name: "Bundee AKi", rating: 7),
new RugbyPlayer(name: "Rory Best", rating: 8),
new RugbyPlayer(name: "Jacob StockDale", rating: 8)
]</pre>
</div>
<h3 style="text-align: left;">
Find a specific record</h3>
<h4 style="text-align: left;">
Java</h4>
<pre class="brush: java; ruler: true; first-line: 1">// Find Tadgh Furlong
Optional<RugbyPlayer> result = players.stream()
.filter(player -> player.getName().indexOf("Tadgh") >= 0)
.findFirst();
String outputMessage = result.isPresent() ? result.get().toString() : "not found";
</pre>
<pre class="brush: java; ruler: true; first-line: 1">System.out.println(outputMessage);</pre>
<br />
<h4 style="text-align: left;">
Groovy</h4>
<pre class="brush: groovy; ruler: true; first-line: 1">println players.find{it.name.indexOf("Tadgh") >= 0}
</pre>
<h4 style="text-align: left;">
Comments</h4>
<ol style="text-align: left;">
</ol>
<ul style="text-align: left;">
<li>The Java lambda has just one parameter: <i>player</i>. This doesn't need to be typed as its type can be inferred. Note: If there were two parameters in the <i>parameter list</i>, parenthesis would be needed around the <i>parameter list</i>.</li>
<li>In Java, a stream must be created from the List first before the functional operation can be applied. </li>
<li>A lambda is then used to before performing a function which returns an Optional</li>
<li>The lambda definition doesn't need a return statement. It also doesn't need {} braces or one of those semi-colons to complete a Java statement. However, you can use {} if you want to and if you want to, you must include the ; and the return statement. Note: if you lambda is more than one line, you don't get a choice, you must use {}. It is recommended <a href="http://www.baeldung.com/java-8-lambda-expressions-tips">best practise</a> to keep Lambda's short and to just one line. </li>
<li>Java 8 supports fluent APIs for pipeline Stream operations. This is also supported in Groovy Collection operations.</li>
<li>In Java a player variable is specified for the Lambda. The Groovy closure doesn't need to specify a variable. It can just use "it" which is the implicit reference to the parameter (similar to _ in Scala). </li>
<li>The <a href="https://docs.oracle.com/javase/8/docs/api/index.html?java/lang/FunctionalInterface.html">Java filter API</a> takes a parameters of type <a href="https://docs.oracle.com/javase/8/docs/api/index.html?java/lang/FunctionalInterface.html">Predicate</a>. A Functional Interface means: <i>can be used as the assignment target for a lambda expression or method reference. </i>Predicate, is type of Functional interface. It's one abstract method is: boolean test(T t). In this case, in the lamda, the player corresponds to t. The body definition should evaluate to a true or a false, in our case player.getName().indexOf("Tadgh") will always evaluate to true or false. True corresponds to a match. </li>
<li>Java 8 has other types of Functional Interfaces:</li>
<ul>
<li style="box-sizing: border-box;"><i style="box-sizing: border-box;">Function</i> – it takes one argument and returns a result</li>
<li style="box-sizing: border-box;"><i style="box-sizing: border-box;">Consumer</i> – it takes one argument and returns no result (represents a side effect)</li>
<li style="box-sizing: border-box;"><i style="box-sizing: border-box;">Supplier</i> – it takes not argument and returns a result</li>
<li style="box-sizing: border-box;"><i style="box-sizing: border-box;">Predicate</i> – it takes one argument and returns a boolean</li>
<li style="box-sizing: border-box;"><i style="box-sizing: border-box;">BiFunction</i> – it takes two arguments and returns a result</li>
<li style="box-sizing: border-box;"><i style="box-sizing: border-box;">BinaryOperator</i> – it is similar to a <i style="box-sizing: border-box;">BiFunction</i>, taking two arguments and returning a result. The two arguments and the result are all of the same types</li>
<li style="box-sizing: border-box;"><i style="box-sizing: border-box;">UnaryOperator</i> – it is similar to a <i style="box-sizing: border-box;">Function</i>, taking a single argument and returning a result of the same type</li>
</ul>
<li>Java 8 can infer the type for the lambda input parameters. Note if you have to specify the parameter type, the declaration must be in brackets which adds further verbosity.</li>
<li>Groovy can println directly. No System.out needed, and no need for subsequent braces.</li>
<li>Like Java, Groovy doesn't need the return statement. However, this isn't just for closures, in Groovy it extends to every method. Whatever is evaluated as the last line is automatically returned. </li>
<li>Groovy has no concept of a Functional interface. <i>This can mean if you forget to ensure your last expression is an appropriate boolean expression, you get unexpected results and bugs at runtime.</i></li>
<li>The arrow operator is used in both Groovy and Java to mean effectively the same thing - separating parameter list from body definition. In Groovy it is only needed it you need to declare the parameters (the default it, doesn't suffice). Note: In Scala, => is used.</li>
</ul>
<ol style="text-align: left;">
</ol>
<h3 style="text-align: left;">
</h3>
<h3 style="text-align: left;">
Find specific records</h3>
<h3 style="text-align: left;">
</h3>
<h4 style="text-align: left;">
Java</h4>
<pre class="brush: groovy; ruler: true; first-line: 1">// Find all players with a rating over 8
List<RugbyPlayer> ratedPlayers = players.stream()
.filter(player -> player.getRating() >= 8)
.collect(Collectors.toList());
ratedPlayers.forEach(System.out::println);
</pre>
<br />
<h4 style="text-align: left;">
Groovy</h4>
<pre class="brush: groovy; ruler: true; first-line: 1">println players.findAll{it.rating >= 8}
</pre>
<h4 style="text-align: left;">
Comments</h4>
<ul style="text-align: left;">
<li>In the Java version, the iterable object <i>ratedPlayers</i> has its forEach method invoked. This method takes a FunctionalInterface of type <a href="https://docs.oracle.com/javase/8/docs/api/java/util/function/Consumer.html">Consumer</a> (see Jdoc <a href="https://docs.oracle.com/javase/8/docs/api/java/lang/Iterable.html">here</a>). Consumer, methods a function which takes an input parameter but returns nothing, it is void. </li>
<li>In Java, the stream.filter() will return another stream. Stream.collect() is one of Java 8's Stream terminal methods. It performs mutable fold operations on the data elements held inside the stream instance return by the filter method. </li>
<li>Collectors.toList () returns a Collector which collects all Stream elements into a List.</li>
<li>When using the toList() collector, you can't assume the type of List that will be used. If you want more control you need to use the toCollection(). For example: </li>
<ul>
<li>.collect(toCollection(LinkedList::new)</li>
</ul>
<li><i>Note: We could have omitted the .collect() operation and invoked
forEach straight on the stream. This would make the Java code
shorter. </i></li>
</ul>
<pre class="brush: groovy; ruler: true; first-line: 1">players.stream()
.filter(player -> player.getRating() >= 8)
.forEach(System.out::println);
</pre>
<ul>
<li>System.out::println is a method reference - a new feature in Java 8. It is syntactic sugar to reduce the verbosity of some lambdas. This is essentially saying, for every element in ratedPlayers, execute, System.out.println, passing in the the current element as a parameter.</li>
<li>Again less syntax from Groovy. The function can operate on the collection, there is no need to create a Stream. </li>
<li>We could have just printed the entire list in the Java sample, but <i>heck</i> I wanted to demo forEach and method reference.</li>
</ul>
<h3 style="text-align: left;">
</h3>
<h3 style="text-align: left;">
Map from object type to another</h3>
<h4 style="text-align: left;">
Java</h4>
<pre class="brush: groovy; ruler: true; first-line: 1">// Map the Rugby players to just names.
// Note, the way we convert the list to a stream and then back again to a to a list using the collect API.
System.out.println("Names only...");
List<String> playerNames = players
.stream()
.map(player -> player.getName())
.collect(Collectors.toList());
playerNames.forEach(System.out::println);
</pre>
<br />
<h4 style="text-align: left;">
Groovy:</h4>
<pre class="brush: groovy; ruler: true; first-line: 1">println players.collect{it.name}</pre>
<div style="text-align: left;">
<h4 style="text-align: left;">
Comments</h4>
</div>
<ul style="text-align: left;">
<li>A stream is needed to be created first before executing the Lambda. Then the collect() method is invoked on the Stream - this is needed to convert it back to a List. This makes code more verbose. </li>
</ul>
<ul>
<li>That said, if all you are doing is printing the list, you can just do...
<pre class="brush: groovy; ruler: true; first-line: 1">players.stream()
.map(player -> player.getName())
.forEach(System.out::println);
</pre>
</li>
</ul>
<h3 style="text-align: left;">
</h3>
<h3 style="text-align: left;">
Perform a Reduction calculation</h3>
<h4 style="text-align: left;">
Java</h4>
<pre class="brush: groovy; ruler: true; first-line: 1">System.out.println("Max player rating only...");
Optional<Integer> maxRatingOptional = players.
stream()
.map(RugbyPlayer::getRating)
.reduce(Integer::max);
String maxRating = maxRatingOptional.isPresent() ? maxRatingOptional.get().toString() : "No max";
System.out.println("Max rating=" + maxRating);
</pre>
<br />
<h4 style="text-align: left;">
Groovy</h4>
<pre class="brush: groovy; ruler: true; first-line: 1">def here = players.inject(null){
max, it ->
it.rating > max?.rating ? it : max
} </pre>
<br />
<h4 style="text-align: left;">
Comments</h4>
<ol style="text-align: left;">
</ol>
<ul style="text-align: left;">
<li>In the Java version, the reduce operation is invoked on the Stream. There are <a href="https://docs.oracle.com/javase/8/docs/api/index.html?java/util/stream/BaseStream.html">three different versions</a> of this method. In this version, no initial value is specified meaning and an Optional type is returned. The input parameter of type BinaryOperator. Because BinaryOperator is a functional interface it means a lamda expression or method reference can be used to specify its value. In this case, the method reference Integer.max() is used.</li>
<li>The null safe operator is used in the Groovy inject closure - so that the first comparsion will work </li>
<li>In Java, it is possible to avoid the isPresent check on the optional by just doing...
<pre class="brush: groovy; ruler: true; first-line: 1">
players.stream()
.map(RugbyPlayer::getRating)
.reduce(Integer::max);
.map(Objects::toString).orElse("No max")
</pre>
</ul>
<ol style="text-align: left;">
</ol>
<br />
<h3 style="text-align: left;">
Summary</h3>
<ol style="text-align: left;">
</ol>
<ul style="text-align: left;">
<li>Groovy is still far more terse</li>
<li>However, some of the function operations in Java are lazily run. For example map(), filter() which are considered intermediate. Intermediate operations produce antoher Stream. They won't execute unless a terminal function e.g. forEach, collect, reduce is invoked on the stream. Terminal functions are value or side-effect producing. </li>
<li>Intermediate operations can either be stateless or stateful. Stateless operations like map() or filter() can operate on elements independently. Stateful operations like distinct() or sorted() may incorporate data from previously seen elements. </li>
<li>The elements of a stream are only visited once during the life of a stream. Like an <a href="https://docs.oracle.com/javase/8/docs/api/java/util/Iterator.html" style="background-color: white; color: #4a6782; font-family: "DejaVu Serif", Georgia, "Times New Roman", Times, serif; font-size: 14px; text-decoration-line: none;" title="interface in java.util"><code style="font-family: "DejaVu Sans Mono", monospace; line-height: 1.4em; margin-top: 8px; padding-top: 4px;">Iterator</code></a><span style="background-color: white; color: #474747; font-family: "dejavu serif" , "georgia" , "times new roman" , "times" , serif; font-size: 14px;">, a new stream must be generated to revisit the same elements of the source.</span></li>
<li><span style="background-color: white; color: #474747; font-family: "dejavu serif" , "georgia" , "times new roman" , "times" , serif; font-size: 14px;">In Java p</span>rocessing streams lazily allows for two performances efficiencies:</li>
<ul>
<li>fusing of multiple operations to minimise passes of the data</li>
<li>avoiding examination of the data unless it is necessary. A stream() may have an infinite number of elements, but a findFirst() or limit() operation will mean that only data that needs to checked will be.</li>
</ul>
<li>This may the code more verbose in cases, but it also means it can be more performant.</li>
<li>Groovy also offers some lazy functions. </li>
</ul>
<ol style="text-align: left;">
</ol>
Full Java code <a href="https://www.jdoodle.com/a/Akt">here</a>. Full Groovy code <a href="https://www.jdoodle.com/a/Akw">here</a>.</div>
Alex Staveleyhttp://www.blogger.com/profile/04378245339254378938noreply@blogger.com0tag:blogger.com,1999:blog-8965734277283465944.post-4606120025273253202018-05-12T14:03:00.003-07:002018-05-25T01:25:13.814-07:00Scala Syntax: 7 points<div dir="ltr" style="text-align: left;" trbidi="on">
A <a href="http://dublintech.blogspot.ie/2013/05/courseras-scala-course.html">few years</a> back I dipped into some Scala as a hobby language. Recently, in order to get a quick overview of Spark I did the 'Big Data Analysis with Scala and Spark' from <a href="https://www.coursera.org/learn/scala-spark-big-data">Coursera</a>. It's a great course. But, one aspect I found challenging was just getting my head around Scala syntax again. Some of it, yeah the <i>basic stuff </i>can be counter-intuitive depending on your perspective.<br />
<br />
<h3 style="text-align: left;">
1. Method / Function Definition</h3>
Typing on the right rather than the left. Consider this simple function definition:<br />
<pre class="brush: scala">def sayHello(param: String): String = {
"Hello" + param
}
</pre>
Javaholics will note:<br />
<ul style="text-align: left;">
<li>The return is specified at the end of the method definition, rather than the beginning. </li>
<li>The type of the parameter is specified after the parameter name rather than before. </li>
<li>Before the function body there is a = </li>
<li>There are two colons (:), one between the parameter and the type and one before the return type.</li>
</ul>
<h3 style="text-align: left;">
2. Unit</h3>
Google "Unit" and you be quickly told you that Unit is the Scala's version of Java void. But, Java’s void is a <i>keyword</i>. Scala’s Unit is a final class which only has one value: () - which is like an alias for no information. Unit indicates a method returns nothing and therefore has side effects, something we don't want to do much of in Scala. So is that counter intuitive? No.<br />
But here is what I find is. If a function has no return type in the function definition and no equals it means Unit is implicitly the return type. Example:<br />
<pre class="brush: scala">def procedure {
println "This String is not returned"
}
procedure: ()Unit
</pre>
Big deal? Of course not. But what about:<br />
<pre class="brush: scala">def procedure {
"This String is not returned"
}
</pre>
Expect the String to be returned, it wont be. How about this?<br />
<pre class="brush: scala">def addNumbers(a: Integer, b: Integer) {
return a + b
}
</pre>
This will give a compile warning: <console>:12: warning: enclosing method addNumbers has result type Unit: return value discarded return a + b It will compile but nothing will be returned: </console><br />
<pre class="brush: scala">def addNumbers(a: Integer, b: Integer) {
a + b
}
</pre>
will give no compile warning and will also return nothing.<br />
<h3 style="text-align: left;">
3. Underscore</h3>
In anonymous Scala functions, _ is like Groovy's <i>it</i>. In Groovy we can to multiple all numbers between 1 and 5 we can do:<br />
<pre class="brush: groovy">(1..5).collect {it * 2}
</pre>
In Scala we can do:<br />
<pre class="brush: scala">(1 to 5).map{_*2}
</pre>
However, in Scala, the second time _ is referenced, it refers to the second parameter<br />
<pre class="brush: scala">val ns = List(1, 2, 3, 4)
val s0 = ns.foldLeft (0) (_+_) //10
</pre>
<h3 style="text-align: left;">
4. Passing anonymous functions. </h3>
Pass one anonymous function and you don't need any curly parenthesis. Pass two and you do.<br />
<pre class="brush: scala">def compose(g:R=>R, h:R=>R) = (x:R) => g(h(x))
val f = compose({_*2}, {_-1})
</pre>
<br />
<h3 style="text-align: left;">
5. Arity-0 </h3>
When a method has no arguments, (arity-0), the parentheses can be omitted in invocation<br />
<pre class="brush: scala">size()
...
size // do it like this
</pre>
But this technique should never be used when method has side effects. So,<br />
<pre class="brush: scala">queue.size // ok
println // not ok do println()
</pre>
<h3 style="text-align: left;">
6. Declare parameter types</h3>
Function defiinitions / Method definition have to declare parameter types but function literals don’t.<br />
<pre class="brush: scala">def addNumbers(a, b): Number {
<console>:1: error: ':' expected but ',' found.
</console></pre>
<h3 style="text-align: left;">
7. Ternary Operator</h3>
There is no ternary operator in Scala. There is one in Java, Groovy, JavaScript. Python 2.5 added support for it. Instead you can do if else on one line and since if / else is an expression you can return a value. For example: In Java we would do:<br />
<pre class="brush: java">(eurovision.winner == "Ireland") ? "Yippee" : "It's a fix"
</pre>
Scala, it's:<br />
<pre class="brush: scala">if (eurovision.winner == "Ireland") "Yippee" else "It's a fix"
</pre>
</div>
Alex Staveleyhttp://www.blogger.com/profile/04378245339254378938noreply@blogger.com0tag:blogger.com,1999:blog-8965734277283465944.post-68385123740102222872018-05-11T16:46:00.003-07:002018-05-11T16:46:48.424-07:00And some more REST tips<div dir="ltr" style="text-align: left;" trbidi="on">
In previous blog <a href="http://dublintech.blogspot.ie/2018/01/when-rest-resource-should-get-its-own.html">posts</a> I have covered some ideas and tips for achieving a REST architecture. In this post, I cover a few more ideas and tips. <br />
<h3 style="text-align: left;">
Caching</h3>
<ul style="text-align: left;">
<li>Caching is a big part of the original dissertation. See <a href="https://www.ics.uci.edu/~fielding/pubs/dissertation/rest_arch_style.htm">section 5.1.4 </a></li>
<li>Strategies include validation (<i>client checks it has the latest version</i>) and expiration (<i>client assumes it has the latest version until a specified time</i>)</li>
<li>Expiration:</li>
<ul>
<li>Expires header tells client when resource is going to expire. The value 0 means avoid caching</li>
</ul>
<ul>
<li>Cache-Control</li>
<ul>
<li>Use max-age directive to specify how long response should be considered valid for; s-maxage for shared caches </li>
<li>Can also be used in requests no-cache means re validate response with server</li>
</ul>
</ul>
<li>Validation </li>
<ul>
<li>Etag - unique version of resource. Used in conjunction with If-none-match request header</li>
<li>Last-Modified - tells client when resource last changed</li>
</ul>
</ul>
<h3 style="text-align: left;">
Controller APIs</h3>
<ul style="text-align: left;">
<li>When something does fit neatly to a CRUD operation, consider a <a href="http://dublintech.blogspot.ie/2018/05/rest-useing-controller-endpoint.html">Controller API</a></li>
</ul>
<h3 style="text-align: left;">
Handling Dates
</h3>
<ul>
<li>
Use <a href="https://en.wikipedia.org/wiki/ISO_8601">ISO-8601</a> for your dates - better for natural sorting, handles timezone, locale nuetral, support from most programming languages</li>
<li>Accept any timezone as anyone in the world may call your API</li>
<li>Store in <a href="https://www.timeanddate.com/time/aboututc.html">UTC</a>, not in your server's timezone. There should be no offset when persisted.</li>
<li>Return in UTC. Allow the client to adjust to its timezone as necessary</li>
<li>Don't use time if you don't need it. If Date only suffices, only persist Date. This means, timezone complexity goes away. </li>
</ul>
<h3 style="text-align: left;">
HEAD</h3>
<ul style="text-align: left;">
<li><a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/HEAD">HEAD action</a> should return response headers </li>
</ul>
<h3 style="text-align: left;">
Headers</h3>
<ul style="text-align: left;">
<li>Always return what headers are useful. Consider: </li>
<ul>
<li>Content-Type</li>
<li>Content-Length</li>
<li>Last-Modified</li>
<li>ETag</li>
<li>Location </li>
</ul>
</ul>
<h3 style="text-align: left;">
Hypermedia (advantages)</h3>
<ul style="text-align: left;">
<li style="margin: 0px !important;">Less coupling</li>
<li style="margin: 0px !important;">Consistent format for links => cleaner client code</li>
<li style="margin: 0px !important;">Developer productivity: API's easier to navigate </li>
<li style="margin: 0px !important;">Make easier to introduce services in a more granular way</li>
<li style="margin: 0px !important;">Code easier to debug - messages always have the URL that created them via the self link</li>
</ul>
<h3 style="text-align: left;">
Hypermedia (choices)</h3>
<ul style="text-align: left;">
<li><a href="http://stateless.co/hal_specification.html">HAL</a> - reduces Address coupling </li>
<li><a href="https://github.com/kevinswiber/siren">SIREN</a> - reduces Address and Actions coupling</li>
<li><a href="http://amundsen.com/media-types/collection/">Collection+JSON </a>(CJ) - reduces Address, Action and Object coupling</li>
</ul>
<h3 style="text-align: left;">
Idempotent</h3>
<ul style="text-align: left;">
<li>Can be called several times and return the same result</li>
<li>OPTIONS, GET, HEAD, PUT and DELETE are all idempotent</li>
</ul>
<h3 style="text-align: left;">
Long Running Requests</h3>
<ul style="text-align: left;">
<li>Some operations take a long time. In such cases, consider returning a 202 with the location field set to a URL the client can poll to check for operation progress. </li>
</ul>
<h3 style="text-align: left;">
Method not allowed</h3>
<ul style="text-align: left;">
<li>If an API only supports GET, it should return a <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/405">405</a> for any PUT, POST, DELETEs etc</li>
</ul>
<div>
<h3 style="text-align: left;">
Must Ignore Principle</h3>
<ul style="text-align: left;">
<li>Clients should ignore data they are not interested in. This makes it much easier for APIs to be backwardly compatible . If an API returns extra data and some clients aren't expecting it they will just ignore it. </li>
</ul>
<h3 style="text-align: left;">
Not acceptable</h3>
<ul style="text-align: left;">
<li style="margin: 0px !important;">When a resource doesn't support a specific media type, it should return 406 (s<em style="font-style: italic;">ee Masse, Rule: 406 (“Not Acceptable”) must be used when the requested media type cannot be served</em></li>
</ul>
<h3 style="text-align: left;">
OPTIONS</h3>
<ul style="text-align: left;">
<li>OPTIONS should return what actions are available on a resource </li>
</ul>
<h3 style="text-align: left;">
Partial Update</h3>
<ul style="text-align: left;">
<li>Handle partial updates with <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/PATCH">PATCH</a></li>
</ul>
<h3 style="text-align: left;">
Query</h3>
<ul style="text-align: left;">
<li>The query component of a URI should be used to filter collections</li>
</ul>
<h3 style="text-align: left;">
Resource Creation</h3>
<ul style="text-align: left;">
<li>When a Resource has been successfully created a <a href="https://httpstatuses.com/201">201</a> should be returned </li>
<li>The <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Location">location header</a> should indicate the URL to get the Resource. </li>
</ul>
<h3 style="text-align: left;">
Safe</h3>
<ul style="text-align: left;">
<li>Actions are considered <i>Safe</i> if they Do not modify resources</li>
<li>OPTIONS, GET and HEAD are safe </li>
</ul>
<h3 style="text-align: left;">
Self link</h3>
<ul style="text-align: left;">
<li>Response bodies should always include a self link - the URL that was used to return the resource. </li>
</ul>
<h3 style="text-align: left;">
Singular or Plural?</h3>
<ul style="text-align: left;">
<li>Use Singular for Singular Document type resource - when there can only be one. For example: /humans/12343343/head</li>
<li>Otherwise plural</li>
</ul>
<ul>
</ul>
</div>
</div>
Alex Staveleyhttp://www.blogger.com/profile/04378245339254378938noreply@blogger.com0tag:blogger.com,1999:blog-8965734277283465944.post-52344393245990991372018-05-11T14:32:00.001-07:002018-05-14T10:31:41.373-07:00REST: Using a Controller endpoint? <div dir="ltr" style="text-align: left;" trbidi="on">
In REST architectures, the fundamental concept is a Resource. A Resource represents <i></i><i>anything that’s important enough to be referenced as a
thing in itself</i>. For example, a Shopping Cart, a Book or a Car. The next fundamental concept is the <i><a href="https://stackoverflow.com/questions/25172600/rest-what-exactly-is-meant-by-uniform-interface">Uniform Interface</a> </i>for accessing and manipulating the Resources. In HTTP land usually means:<br />
<ul style="text-align: left;">
<li>Create is POST </li>
<li>Read is GET </li>
<li>Update is PUT (or PATCH for Partial Update) </li>
<li>Delete is DELETE</li>
</ul>
There are of course other concepts (statelessness, caching etc) but for this blog post, let's just focus on Resources. <br />
<br />
In the real world, many <i>things</i> map nicely to Resources. However, inevitably somethings won't map so nicely to resources. This is usually a minority of operations for example <i>reset password</i>. It's possible to model these as either<br />
<ul style="text-align: left;">
<li> a PUT on /password/ </li>
</ul>
or as<br />
<ul style="text-align: left;">
<li> a <i>Controller endpoint</i> and a POST to /resetpassword </li>
</ul>
The latter may be considered to be closer to programmatic REST than pure REST, but there are times when clients and customers will want you to be pragmatic.
This article gives suggestions regarding when to consider using the Controller option.<br />
<br />
<h3 style="text-align: left;">
Does the action Map to a CRUD? </h3>
<div style="text-align: left;">
Several actions in a real world application will not map nicely to a Create Read Update Delete (CRUD). For example, Paypal's cancel billing agreement API is:
</div>
<pre class="brush: text; ruler: true; first-line: 1">POST /v1/payments/billing-agreements/agreement_id/cancel
</pre>
The cancel action rarely maps nicely to a CRUD for a resource. It could be interpreted as:<br />
<ul style="text-align: left;">
<li>some resource gets be created (A cancel record) </li>
<li>some resource gets updated (some status column could be getting set to cancelled) </li>
<li>or some resource gets deleted (a order request gets deleted). </li>
</ul>
Why should the client have to care about how cancel is handled? Couldn't it always change?
In some case API's have got around the doesn't map nicely to a CRUD problem using HTTP tunneling. For <i>cancelling a billing agreement</i> this would like:
<br />
<pre class="brush: text; ruler: true; first-line: 1">POST /v1/payments/billing-agreements/agreement_id
</pre>
with body:
<br />
<pre class="brush: text; ruler: true; first-line: 1">{
"operation":"cancel"
}
</pre>
This is considered an anti-pattern and should never be used.
Instead a Controller end point should be used.<br />
<h3 style="text-align: left;">
</h3>
<h3 style="text-align: left;">
Resource State or Workflow? </h3>
In a REST architecture, every request between Client or Server will usually change a Resource State (<i>write operation</i>) or the Application State (<i>a query or read operation</i>). However, in the real world workflows are inevitable. For example, a reset password flow usually consists of:<br />
<ul style="text-align: left;">
<li>Asking the user for the userId (usually email) </li>
<li>System checking that email exists on the system </li>
<li>Sending the user an email with a link to reset the password </li>
<li>Ensuring the user only has a set amount of time to click the link </li>
<li>When the user clicks the link they may be asked a bunch of questions </li>
<li>They will be asked to retype their new password to ensure there's no typos </li>
</ul>
When an client action is part of a complex workflow, Resource state and Application state changes may not be easy to model.
They may not happen synchronously and they could change based on how the workflow is modelled or when the workflow needs to add an extra step.
In such scenarios, consider using a Controller end point.<br />
<h3 style="text-align: left;">
</h3>
<h3 style="text-align: left;">
REST without PUT </h3>
<div style="text-align: left;">
In some situations, arguments can be made for avoiding PUT and instead using POST to a different endpoint which signifies intent.
For example, to change address instead of invoking a PUT to /address/, the client would invoke a POST to /changeaddress and avoid PUTs altogether. One example where this approach is useful is when handling asynchronous operations and you are trying to make clear atomic consistent operation. So for example, if changing address takes a long time and you would rather return a 202, with a location field for the client to poll, if you use the /changeaddress you can then leave /address endpoints as those that are only atomically consistent.<br />
<br />
So, any PUT or POST to address, means if you were to immediately do a GET you would get the consistent view of the Resource. This approach is also useful if you want to model the Business event rather than the actual resource that is changing. So for example, suppose 6 or 7 things need to take place when a Bank account has been closed for a Business process perspective. All on the back end in the same thread / transaction. Again, here POST to controller endpoint such as /accountclosed makes more sense then /DELETE to /account. <br />
<br />
See this <a href="https://www.thoughtworks.com/insights/blog/rest-api-design-resource-modeling">article</a> for more info. <br />
<br />
<h3 style="text-align: left;">
Summary </h3>
</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
So why there may be subjectivity involved on when to use a controller style endpoint. The above may at least help to you to make a decision. Remember, it should always only be a minority of APIs where you consider this approach. You are outside the conventional Uniform Interface for unique style operations but you want to still make them feel intuitive to clients of the API. </div>
</div>
Alex Staveleyhttp://www.blogger.com/profile/04378245339254378938noreply@blogger.com1tag:blogger.com,1999:blog-8965734277283465944.post-36783517300802082212018-02-27T15:21:00.001-08:002018-02-28T06:34:30.850-08:00Testing your code with Spock<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<a href="http://spockframework.org/">Spock</a> is a testing and specification framework for Java and Groovy applications.
Spock is:<br />
<ul style="text-align: left;">
<li>extremely expressive </li>
<li>facilitates the Given / When / Then <a href="https://martinfowler.com/bliki/GivenWhenThen.html">syntax</a> for your tests </li>
<li>compatible with most IDEs and CI Servers.</li>
</ul>
Sounds interesting? Well you can start playing with Spock very quickly by paying a quick visit to the
<a href="http://meetspock.appspot.com/?id=9001">Spock web console</a>. When you have a little test you like, you can publish
it like I did for this little <a href="http://meetspock.appspot.com/script/5071001851265024">Hello World test</a>.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg85IlD4F0iOYHhf7RjDbu24jEDy_4NWKAQR-MUpYRGH8rEfC5pn4BgwZ27J4LY_ogoJeNuZk9Qymvac8Q82F_taeils5v7QPf52m_3f4MOON5hvQOBvrlKkcd91Jzo-k1G4RPdgSCpETVd/s1600/Screen+Shot+2018-02-28+at+14.27.42.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="996" data-original-width="1330" height="478" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg85IlD4F0iOYHhf7RjDbu24jEDy_4NWKAQR-MUpYRGH8rEfC5pn4BgwZ27J4LY_ogoJeNuZk9Qymvac8Q82F_taeils5v7QPf52m_3f4MOON5hvQOBvrlKkcd91Jzo-k1G4RPdgSCpETVd/s640/Screen+Shot+2018-02-28+at+14.27.42.png" width="640" /></a></div>
<br />
This Hello World test serves as a gentle introduction to some of the features of Spock.
<br />
<br />
Firstly, Spock tests are written in Groovy. That means, some boiler plate code that you have with Java goes away. There is<br />
<ul style="text-align: left;">
<li><i>No need</i> to indicate the class is Public as it is by default.</li>
<li><i>No need</i> to declare firstWord and lastWord as Strings </li>
<li><i>No need</i> to hurt your little finger with a ; at the end every line</li>
<li><i>No need
</i>to explicitly invoke assert, as every line of code in the expect block gets that automatically. <i>Just make
sure the lines in the then: block evaluate to a boolean expression. If it is true the test passes otherwise it fails. So in this case, it is just an equality expression which will either be
true or false. You can have as many expressions as you want. </i></li>
</ul>
So less boiler plate code what next? Well you know those really long test names you get with JUnit tests,
well instead of having to call this test, <i>helloWorldIntroductionToSpockTest</i>() which is difficult to read, you can just
use a String with spaces to name the test: <i>Hello World introduction to Spock test</i>. This makes things much more readable.<br />
<br />
Thirdly, the <i>Given: When: Then:</i> syntax, enforces test structure. No random asserts all the test. They are in a designated place. More complex tests, can use this structure to achieve <a href="https://en.wikipedia.org/wiki/Behavior-driven_development">BDD</a> and <a href="https://www.agilealliance.org/glossary/atdd/">ATDD</a>. <br />
<br />
Fourthly, if I were to make a small change to the test and change the assertion to also include Tony, the test will of course fail.
But when I get a failure in Spock, I get the full context of the expression that is tested. I see the value of everything in the expression. This makes it much quicker to diagnose problems when tests fail.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjve91_tJUbv_Pha9r7vfA3ZZLwDOIOda5ja0ntiEywQoMbOi7JNpvL6Toi07rUPxedtKQUobUfEeaHXgcyO5YFz8t8nzXAXjzrhRx54u2BbiuqDSJM9SMMLYbHt32s5NRfKLlGAZ1IXxC_/s1600/Screen+Shot+2018-02-28+at+14.26.25.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1340" data-original-width="1056" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjve91_tJUbv_Pha9r7vfA3ZZLwDOIOda5ja0ntiEywQoMbOi7JNpvL6Toi07rUPxedtKQUobUfEeaHXgcyO5YFz8t8nzXAXjzrhRx54u2BbiuqDSJM9SMMLYbHt32s5NRfKLlGAZ1IXxC_/s640/Screen+Shot+2018-02-28+at+14.26.25.png" width="504" /></a></div>
<br />
<br />
Not bad for an introduction. Let's now have a look at more features. <br />
<br />
<h4 style="text-align: left;">
Mocking and Stubbing</h4>
Mocking and Stubbing are much more powerful than what is possible with JUnit (<i>and various add on's</i>). But, it is not only super powerful in Spock,
it is also very terse, keeping your test code very neat and easy to read.<br />
<br />
Suppose we want to Stub a class called <i>PaymentCalculator</i> in
our test, more specifically one of its method, <i>calculate(Product product, Integer count).</i> In the stubbed version
we want to return the count multiplied by 10 irrespective of the value of product. In Spock we achieve this by:
<br />
<pre class="brush: groovy; ruler: true; first-line: 1">PaymentCalculator paymentCalculator = Stub(PaymentCalculator)
paymentCalculator.calculate(_, _) >> {p, c -> c * 10}
</pre>
If you haven't realised how short and neat this is, well then get yourself a <a href="https://www.bewleys.com/ie/coffee-project/">coffee</a>. If you have realised well you can still have a coffer but consider these points:<br />
<ol style="text-align: left;">
<li>The underscores in the calculate mean for all values </li>
<li>On the right hand side, of the second line, we see a Groovy Closure. For now, think of this as an anonymous method with two inputs.
<i>p</i> for the <i>product</i>, <i>c</i> for <i>count</i>. We don't have to type them. That's just more boiler plate code gone. </li>
<li>The closure will always return the count time 10. We don't need a <i>return</i> statement. The value of the last expression is always
returned. Again, this means less boiler plate code.
When stubbing becomes this easy and neat, it means you can really focus on the test - cool. </li>
</ol>
<h4 style="text-align: left;">
Parameterised Tests </h4>
The best way to explain this is by example.
<br />
<pre class="brush: groovy; ruler: true; first-line: 1">@Unroll
def "Check that the rugby player #player who has Irish status #isIrish plays for Ireland"(String player, Boolean isIrish) {
given:"An instance of Rugby player validator"
RugbyPlayerValidator rugbyPlayerValidator = new RugbyPlayerValidator()
expect:
rugbyPlayerValidator.isIrish(player) == isIrish
where:
player || isIrish
"Johny Sexton" || true
"Stuart Hogg" || false
"Conor Murray" || true
"George North" || false
"Jack Nowell" || true
}</pre>
In this parameterised test we see the following:<br />
<ol style="text-align: left;">
<li>The test is parameterised. The test signature having parameters tells use this, as do the <i>where</i> block. </li>
<li>There is one input parameter player and one output parameter
- which corresponds to an expected value. </li>
<li>The test is parameterised five times. The input parameters are on the left, output on the right.
It is, of course, possible to have more of either, in this test we just have one of each. </li>
<li>The @Unroll annotation will mean that if the test fails, the
values of all parameters will be outputted. The message will substitute the details
of player into #player and the details of the Irish status substituted into #isIrish. So for example, "<i>Checks that the rugby player Jack Nowell who has Irish status true plays for Ireland</i>"</li>
</ol>
Again, this makes it much quicker to narrow in on bugs. Is the test wrong or is the code wrong? That becomes a question that can be answered faster. In this case, of course it is the test that is wrong.<br />
<br />
<h4 style="text-align: left;">
All the benefits of Groovy</h4>
What else?
Well another major benefit is all the benefits of Groovy. For example, if you are testing an API that returns JSON or XML, Groovy is brilliant for parsing XML and JSON. Suppose we have an API that returns information about sports players in XML format.
The format varies, but only slightly, depending on the sport they play:
<br />
<pre class="brush: xml; ruler: true; first-line: 1"><details>
<rugbysummarycategory>
<players>
<player>Joey Carberry</player>
<player>Teddy Thomas</player>
</players>
</rugbysummarycategory>
</details>
<details>
<footballsummarycategory>
<players>
<player>Lionel Messi</player>
<player>Cristiano Ronaldo</player>
</players>
</footballsummarycategory>
</details>
</pre>
We want to just invoke this API and then parse out the players irrespective of the sport.
We can parse this polymorphically very simply in Groovy.
<br />
<pre class="brush: groovy; ruler: true; first-line: 1">def rootNode = new XmlSlurper().parseText(xml)
def players = rootNode.'*'.Players.Player*.text()
</pre>
<br />
Some key points:<br />
<ol style="text-align: left;">
<li>The power of dynamic typing is immediate. The expression can be dynamically invoked on the rootNode.
No verbose, complex XPath expression needed.</li>
<li>The '*', is like a wildcard. That will cover both RugbySummaryCategory and FootballSummaryCategory.</li>
<li>The Player*, means for all Player elements. So no silly verbose for loop needed here </li>
<li>The text() expression just pulls out the values of the text between the respective Player elements.
So why now have a list all players and can simple do:<i>players.size() == 4</i>. Remember, there is no need for the assert. </li>
</ol>
Suppose we want to check the players names. Well in this case we don't care about order, so make more sense to convert the list to a Set and then check. Simple.
<br />
<pre class="brush: xml; ruler: true; first-line: 1">players as Set == ["Joey Carberry", "Teddy Thomas", "Lionel Messi", Cristiano Ranaldo"] as Set</pre>
<br />
This will convert both list to a Set which means then order checking is gone and it is just a Set comparison.
There's a tonne more Groovy features we can take advantage of. But the beauty is, we don't actually have to.
<b><i>All Java code is also valid in a Groovy class</i></b>. The same hold trues for Spock. This means there is no steep learner curve
for anyone from a Java background. They can code pure Java and then get some Groovy tips from code reviews etc.<br />
<h4 style="text-align: left;">
Powerful annotations </h4>
Spock also has a range of powerful annotations for your tests. Again, we see the power of Groovy here as we can pass a closure
to these annotations. For example:
<br />
<pre class="brush: groovy; ruler: true; first-line: 1">@IgnoreIf({System.getProperty("os.name").contains("windows")})
def "I'll run anywhere except windows"() {...}
</pre>
Or just make your test fail if they take too long to execute
<br />
<pre class="brush: groovy; ruler: true; first-line: 1">@Timeout(value = 100, unit=TimeUnit.MILLISECONDS)
def "I better be quick"() {...}
</pre>
So in summary Spock versus vanilla JUnit has the following advantages:<br />
<ol style="text-align: left;">
<li>Test Structure enforced. No more random asserts. Assertions can only be in designated parts of the code. </li>
<li>Test code is much more readable. </li>
<li>Much more information on the context of the failed test</li>
<li>Can mock and stub with much less code</li>
<li>Can leverage a pile of Groovy features to make code much less verbose</li>
<li>Very powerful test parameterisation which can be done very neatly</li>
<li>A range of powerful annotations. </li>
</ol>
And one of the often forgotten points is that your project doesn't have to be written in Groovy.
You can keep it all in Java and leverage the static typing of Java for your production code and use the power and speed of Groovy for your test code.<br />
<br />
Until the next time take care of yourselves.
</div>
Alex Staveleyhttp://www.blogger.com/profile/04378245339254378938noreply@blogger.com0tag:blogger.com,1999:blog-8965734277283465944.post-33035455333616550802018-01-12T09:52:00.002-08:002018-01-12T09:54:37.525-08:00When a REST Resource should get its own Address?<div dir="ltr" style="text-align: left;" trbidi="on">
<h2 style="text-align: left;">
Background </h2>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhRw7Le-3DaQW22FjAoqm3Uu7MpKaBPKvtDji0tu5vi1n70n2R5GanRPwPndpS1K-Sce5skAJOTUNacwXJsbCxiRzAMkIDqjD62aAIytvR5xGCMrzNEGkPLucP8z1syFgC1EXFH4BsSv7y7/s1600/index.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="175" data-original-width="288" height="121" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhRw7Le-3DaQW22FjAoqm3Uu7MpKaBPKvtDji0tu5vi1n70n2R5GanRPwPndpS1K-Sce5skAJOTUNacwXJsbCxiRzAMkIDqjD62aAIytvR5xGCMrzNEGkPLucP8z1syFgC1EXFH4BsSv7y7/s200/index.png" width="200" /></a></div>
<h4 style="text-align: left;">
<i>Author's note </i></h4>
<i>In a purist REST approach, all endpoints (except the starting endpoint) are opaque and their various details shouldn't need to be published. Even if this approach is being used, the points in this article are relevant as Server logic will have to determine when something requires a end point or not. </i><br />
<h4 style="text-align: left;">
<i>Introduction </i> </h4>
In a REST architecture an entity or a resource (<i>for the rest of the article the term entity will be used)</i> may or may not have its own address.
For example, suppose we have an inventory application merchants use to sell their products. Immediately it is possible to see a Product entity. It's URL will look something like: /product/{id}<br />
<br />
Now, it is possible for the merchant selling the Products to add his / her own comments to the Products. For example, "<i>Sells very well on Fridays</i>" or "<i>Consider changing price if product doesn't start selling</i>". A Product can have 0..* Comments.
As stated, the Product has its own address: /product/{id} for example /product/1231233<br />
<br />
and a response payload like this
<br />
<pre class="brush: java; text: true; first-line: 1">{
"id":"1231233",
"type":"Beer",
"comments": [{
"id":"1",
"comment":"Sells very well on Fridays"
}, {
"id":"2",
"comment":"Consider changing price if product doesn't start selling"
}]
}</pre>
As can be seen, the payload returns a collection of Comment Objects. Should the individual comments each have their own address as well or is it okay that they are just embedded into the Product response?
To help the answer this question the following should be considered.<br />
<h3 style="text-align: left;">
Does the Entity have any meaning outside the Containing Entity Context? </h3>
If an Entity (for example Comment) has meaning outside their containing Entity (for example Product) then they should have their own address. For example, suppose the Entity was Student and the Student returned a list of Universities he / she had studied. These Universities have their own meaning outside the Student. So obviously the University should have its own address.
In the Activity / Comments scenario, the Comments only exist for the activity. No other Entity will ever reference them or need to reference them. Therefore further aspects needs to be considered.<br />
<h3 style="text-align: left;">
Is it desirable to perform actions on the individual entities? </h3>
Should the client be allowed to create, read, update or delete the individual entity? These have to be considered separately. <br />
<h4 style="text-align: left;">
Writes: Create, Update, Delete </h4>
In the Product / Comments scenario, a Comment would never be created outside or without an Product. It is essentially added to an Product. This could be considered as a partial update to the Product. However, an update or delete to an existing Comment could also be considered a partial update to the Product. This creates complexity on how to differentiate between Create / Updates and Deletes of a Comment using a partial update on the Product. If this is required, it would be much simpler to create a contextual address for the Comment (which indicates the hierarchical nature of the Product / Comment) then allow the Client sent POST, PUT, PATCH, DELETES to that.<br />
<br />
Example URL: /product/1231233/comment/1 <br />
<h4 style="text-align: left;">
Reads </h4>
In some scenarios the parent containing Entity may not return all the information about the child Entities. For example, again consider the Product --> Comment scenario. Suppose the comment was very large. This would mean the payload for the Product was also very large. In such cases, it might be more prudent for the Product to just return a summary of the Comment and if the client wants the full Entity to make an individual request.
Similarly, if there's a big performance cost to get an individual Entity (for example a 3rd party API has to be invoked to get all the information about the comment), it can make more sense to just send a URL link to the Entity rather the than the actual entity contents. <br />
<h4 style="text-align: left;">
N+1 Problem </h4>
If individual Reads are required, be careful that the N+1 problem doesn't then get introduced. For example, suppose a Product could have 100 Comments. the Product API will only return a summary of the Comment and a link to each individual comment if the client wants all the information. However, if the client wants every single comment, this means there will now be 100 HTTP requests. If this is a potential scenario, then a secondary endpoint which aggregates all the comments into the Product should be considered. This is similar to the API Gateway pattern. <br />
<h4 style="text-align: left;">
Surface Area of Endpoints </h4>
In any architecture when contracts are published, if there are too many it can become very unwieldy for developers to understand. Most well known APIs (e.g. PayPal, Amazon, Twitter, Google) usually only have about 20 - 30 addresses. This is a good aim to have. If there are 5,000 different addresses it can become way too large and difficult to control etc.<br />
<br />
In summary, the decision diagram provides guidance on what you should do.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjnW_uj4AX4LbrvvPNqlxUZCnglvY7cLGUB5Mt8451aq2aBvrY9YXBDRB-WZ_kWMb5f6viTlcmELtWBD0q1nocqOT22dq94pofOGZjkvN0cdM7MLwlVFCRR42ADaP1hViUAjqNpiFNlXl7Y/s1600/decision.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1555" data-original-width="666" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjnW_uj4AX4LbrvvPNqlxUZCnglvY7cLGUB5Mt8451aq2aBvrY9YXBDRB-WZ_kWMb5f6viTlcmELtWBD0q1nocqOT22dq94pofOGZjkvN0cdM7MLwlVFCRR42ADaP1hViUAjqNpiFNlXl7Y/s640/decision.png" width="273" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
</div>
Alex Staveleyhttp://www.blogger.com/profile/04378245339254378938noreply@blogger.com0tag:blogger.com,1999:blog-8965734277283465944.post-32978387575733338332018-01-07T14:01:00.003-08:002018-01-13T10:19:15.778-08:00Are you forgetting your Agile values?<div dir="ltr" style="text-align: left;" trbidi="on">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: small;"><span style="color: black; display: inline; float: none; font-style: normal; font-weight: normal; letter-spacing: normal; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">A while back I wrote why sometimes <a href="http://dublintech.blogspot.ie/2016/06/why-agile-can-fail.html">Agile will fail.</a> In this post, I will focus on the specific misunderstandings of Agile values. When people ask if you're Agile, they basically think:</span></span><br />
<ul style="text-align: left;"><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: small;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjgj2jQ1JQkcK_4Yw8vgMYRQ39acoetdnl9abcbOjffJHie_aDkgq8m2rxvsQXyv3s659Z_d_Ukvx9AhIKPnWEXca1kF4UKX1hT0P__DdILQyZZ7m8jKHEuNqPmeG8e5U-AtDBS5NtntR5q/s1600/agile.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="298" data-original-width="300" height="198" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjgj2jQ1JQkcK_4Yw8vgMYRQ39acoetdnl9abcbOjffJHie_aDkgq8m2rxvsQXyv3s659Z_d_Ukvx9AhIKPnWEXca1kF4UKX1hT0P__DdILQyZZ7m8jKHEuNqPmeG8e5U-AtDBS5NtntR5q/s200/agile.png" width="200" /></a></span><span style="font-size: small;">
</span>
<li><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: small;"><span style="color: black; display: inline; float: none; font-style: normal; font-weight: normal; letter-spacing: normal; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">Do you have stand ups?</span></span></li>
<span style="font-size: small;">
</span>
<li><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: small;"><span style="color: black; display: inline; float: none; font-style: normal; font-weight: normal; letter-spacing: normal; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">Do you have retrospectives?</span></span></li>
<span style="font-size: small;">
</span>
<li><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: small;"><span style="color: black; display: inline; float: none; font-style: normal; font-weight: normal; letter-spacing: normal; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">Do you have stories? </span></span></li>
<span style="font-size: small;">
</span>
<li><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: small;"><span style="color: black; display: inline; float: none; font-style: normal; font-weight: normal; letter-spacing: normal; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">Do you use yellow post its? </span></span></li>
<span style="font-size: small;">
</span>
<li><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: small;"><span style="color: black; display: inline; float: none; font-style: normal; font-weight: normal; letter-spacing: normal; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">etc</span></span></li>
</ul>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: small;"><span style="color: black; display: inline; float: none; font-style: normal; font-weight: normal; letter-spacing: normal; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">Such ideas belong to an Agile process called Scrum which is the most popular Agile process but isn't the only one. There are other processes: Kanban, RUP, XP, etc... You don't necessarily have to be Scrum.</span></span><br />
<div style="color: black; font-style: normal; font-weight: normal; letter-spacing: normal; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: small;"><br /></span></div>
<div style="color: black; font-style: normal; font-weight: normal; letter-spacing: normal; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: small;">For me the most important thing that came from Agile was the actual <a href="http://agilemanifesto.org/"><i>manifesto</i></a>. It should be read by everyone working in software at the beginning of every year and at the beginning of every project. Then it should be discussed <span style="font-family: "arial" , "helvetica" , sans-serif;">with</span> team members and people should be encouraged to give specific examples they have of the <i>values</i> and <i>principles</i> detailed in the manifesto. In this blog post, we'll have a look at the values.</span></div>
<div style="color: black; font-style: normal; font-weight: normal; letter-spacing: normal; text-align: left; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">
<h3>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: small;">
8 Values with emphasis on 4</span></h3>
</div>
<div style="color: black; font-style: normal; font-weight: normal; letter-spacing: normal; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: small;">There are four points which detail 8 values in the manifesto</span></div>
<div style="color: black; font-style: normal; font-weight: normal; letter-spacing: normal; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">
<ol style="text-align: left;">
<li><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: small;">Individuals and interactions over processes and tools</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: small;">Working software over comprehensive documentation</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: small;">Customer collaboration over contract negotiation</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: small;">Responding to change over following a plan</span></li>
</ol>
</div>
<div style="color: black; font-style: normal; font-weight: normal; letter-spacing: normal; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">
<div style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: small;">Then there is the very important sentence: "That is, while there is value on items on the right, we value items on the left more." It is worth saying that sentence twice. Maybe three times... four times... whatever it takes so you never forget it. Why? Well let's take them one by one.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: small;"> </span> </div>
<h4 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: small;">
Individuals and interactions over processes and tools </span></h4>
</div>
<div style="color: black; font-style: normal; font-weight: normal; letter-spacing: normal; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: small;">So does this mean if you are Agile do you stop worrying about process? No. No project can ever be successful without process. From an Agile perspective, it means, you value process<span style="font-family: "arial" , "helvetica" , sans-serif;">.</span> <span style="font-family: "arial" , "helvetica" , sans-serif;">B</span>ut, you value Individuals and interactions more. So for example, say you spend lots of man hours in a process that isn't really adding value to the project or customer. When you look at this <span style="font-family: "arial" , "helvetica" , sans-serif;">critically</span> it is because developers and testers don't talk to each other. Instead they have a convoluted process so they think they can blame each other when a production defect happens. Lots of man hours goes into this. But, it would be much more efficient if they talked regularly to each other<span style="font-family: "arial" , "helvetica" , sans-serif;"> which then negates the<span style="font-family: "arial" , "helvetica" , sans-serif;"> </span>asp<span style="font-family: "arial" , "helvetica" , sans-serif;">ects of the process which are making it convoluted</span>. </span>So the challenge is two fold:</span></div>
<div style="color: black; font-size: 12px; font-style: normal; font-weight: normal; letter-spacing: normal; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">
<ul style="text-align: left;"><span style="font-size: small;">
</span>
<li><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: small;">Ensure people talk to each other regularly</span></li>
<span style="font-size: small;">
</span>
<li><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: small;">Ensure your processes are efficient </span></li>
</ul>
</div>
<div style="color: black; font-style: normal; font-weight: normal; letter-spacing: normal; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">
<div style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: small;">So for example, it makes more sense to have a <i>regular</i> <i>show and tell </i>then no interaction whatsoever and a massive delivery at the end where you are hoping you will get customer satisfaction because <span style="font-family: "arial" , "helvetica" , sans-serif;">of so<span style="font-family: "arial" , "helvetica" , sans-serif;">me</span></span> long complex process. Aim to make individuals, interactions in the heart of your processes.</span><br />
</div>
<h4 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: small;">
Working software over comprehensive documentation</span></h4>
</div>
<div style="color: black; font-style: normal; font-weight: normal; letter-spacing: normal; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: small;">This is a classic misunderstanding. Somebody wants comprehensive documentation for some complex functionality and a developer retorts: "Hey Dinasour, this isn't waterfall, there is no need for detailed documentation". Wrong. Documentation is still required. The point here is that with Agile you shouldn't be spending massive amounts of man hours on documentation <i><b>when</b></i> your software is riddled with bugs. It is more important that your software works. This means more time should be spend on excell<span style="font-family: "arial" , "helvetica" , sans-serif;">e</span>nt automated tests that have sufficient functional coverage. This isn't always easy to do but you should ensure this is done. </span><br />
</div>
<div style="color: black; font-style: normal; font-weight: normal; letter-spacing: normal; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">
<h4 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: small;">
Customer collaboration over contract negotiation</span></h4>
</div>
<div style="color: black; font-style: normal; font-weight: normal; letter-spacing: normal; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: small;">Thirdly, do you stop doing contracts agreements with customers on your projects. No. The point here is you spend more man hours with meetings collaborating with customers than you do teasing out a contract with lawyers. Instead of spending a massive amount of time to get sign off on a massive project, you should collaborate thru the life cycle of the project, breaking it up into small increments, take on board feedback and work together towards a common goal: project success. </span><br />
</div>
<div style="color: black; font-style: normal; font-weight: normal; letter-spacing: normal; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">
<h4 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: small;">
Responding to change over following a plan</span></h4>
</div>
<div style="color: black; font-style: normal; font-weight: normal; letter-spacing: normal; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: small;">Lastly, do you stop planning? Of course not. But again, what is the point planning down to the nitty gritty if you cannot even respond to change? Could you imagine a customer asked for a tiny change to how a UI was displaying data and the developer team respond with: "<i>Sorry that wasn't in our 6 month plan</i>"? It is paramount that architecture facilitates reasonable change and if it can't the project will struggle to really embrace an agile philosophy. </span><br />
</div>
<div style="color: black; font-style: normal; font-weight: normal; letter-spacing: normal; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">
<h3 style="text-align: left;">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: small;">
Summary</span></h3>
</div>
<div style="color: black; font-style: normal; font-weight: normal; letter-spacing: normal; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: small;">Agile is actually about putting more <i>constraints</i> on your software methodology. As an analogy, the REST architecture style puts constraints on your architecture for example the Uniform Interface, Statelessness and Caching capabilities. The idea is then by sticking to these constraints (and that's a technical challenge) you get benefits. In the case of REST, your architecture will be more scalable, extensible and lead to much higher developer productivity for API consumers. In the case of Agile, by sticking to the constraints of valuing 8 key concepts but putting even more emphasis on 4, your project will have greater chance of success. </span></div>
<div style="color: black; font-style: normal; font-weight: normal; letter-spacing: normal; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: small;"><br /></span></div>
<div style="color: black; font-style: normal; font-weight: normal; letter-spacing: normal; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: small;"><br /></span></div>
<div style="-webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; color: black; font-family: HelveticaNeue; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<br /></div>
</div>
Alex Staveleyhttp://www.blogger.com/profile/04378245339254378938noreply@blogger.com0tag:blogger.com,1999:blog-8965734277283465944.post-36824501070793021242017-11-15T06:56:00.003-08:002017-11-16T03:57:51.868-08:00More Fail early - Java 8 <div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="text-align: left;" trbidi="on">
<i>Fail fast </i>or <i>Fail early</i> is a software engineering concept that tries to prevent complex problems happening by stopping execution as soon as something that shouldn't happen, happens. In a previous <a href="http://dublintech.blogspot.co.uk/2016/02/fail-safe-fail-fast.html">blog post</a> and <a href="http://dublintech.blogspot.co.uk/2016/02/fail-safe-fail-fast.html">presentation</a> I go more into detail about the merits of this approach, in this blog post I will just detail another use of this idea in Java 8.</div>
In Java, Iterators returned by Collection classes e.g. ArrayList, HashSet, Vector etc are fail fast.
This means, if you try to add() or remove() from the underlying data structure while iterating it
you get a ConcurrentModificationException.
Let's see:
<br />
<pre class="brush: java; ruler: true; first-line: 1">import static java.util.Arrays.asList;
List<Integer> ints = new ArrayList<>(asList(1,2,3,4,5,6,9,15,67,23,22,3,1,4,2));
for (Integer i: ints) {
// some code
ints.add(57); // throws java.util.ConcurrentModificationException
}
</pre>
In Java 8u20, the Collections.sort() API is also fail fast. This means you can't invoke
it inside an iteration either. For example:
<br />
<pre class="brush: java; ruler: true; first-line: 1">import static java.util.Arrays.asList;
List<Integer> ints = new ArrayList<>(asList(1,2,3,4,5,6,9,15,67,23,22,3,1,4,2));
for (Integer i: ints) {
// some code
Collections.sort(ints); // throws java.util.ConcurrentModificationException
}
</pre>
This makes sense. Iterating over a data structure and sorting it during the iteration is
not only counter intuitive but something likely to lead to unpredictable results. Now, you can get away with this and not get the exception if you have break immediately after the sort invocation.
<br />
<pre class="brush: java; ruler: true; first-line: 1">import static java.util.Arrays.asList;
List<integer> ints = new ArrayList<>(asList(1,2,3,4,5,6,9,15,67,23,22,3,1,4,2));
for (Integer i: ints) {
// some code
Collections.sort(ints); // throws java.util.ConcurrentModificationException
break;
}
</pre>
But, that's hardly great code. Try to avoid old skool iterations and you use Lambdas when you can.
But, if you are stuck, just do the sort when outside the iteration
<br />
<pre class="brush: java; ruler: true; first-line: 1">import static java.util.Arrays.asList;
List<integer> ints = new ArrayList<>(asList(1,2,3,4,5,6,9,15,67,23,22,3,1,4,2));
Collections.sort(ints);
for (Integer i: ints) {
// some code
}
</pre>
or use a data structure which sorts when you add.<br />
<br />
This new behaviour of the Collections.sort() API came in <a href="http://www.oracle.com/technetwork/java/javase/8u20-relnotes-2257729.html">Java 8 release 20</a>. It is worth having a look at the specific section that details the change in the API:<br />
"<br />
<b>Area</b>: core-libs/java.util.collections <br />
<b>Synopsis</b>: Collection.sort defers now defers to List.sort<br />
Previously <code>Collection.sort</code>
copied the elements of the list to sort into an array, sorted that
array, then updated list, in place, with those elements in the array,
and the default method <code>List.sort</code> deferred to Collection.sort. This was a non-optimal arrangement.<br />
From 8u20 release onwards <code>Collection.sort</code> defers to <code>List.sort</code>. This means, for example, existing code that calls <code>Collection.sort</code> with an instance of <code>ArrayList</code> will now use the optimal sort implemented by ArrayList.<br />
"<br />
<br />
I think it would have helped if Oracle were a little more explicit here on how this change could cause runtime problems. Considering everybody uses the Collections framework if an API that previously didn't throw a exception now can for the same situation (bad code and all that it is), it is better if the release notes made it easier for developers to find that information out.<br />
<br />
</div>
Alex Staveleyhttp://www.blogger.com/profile/04378245339254378938noreply@blogger.com0tag:blogger.com,1999:blog-8965734277283465944.post-77937902077340956122017-10-05T12:31:00.004-07:002017-10-05T12:40:42.941-07:00Book Review: RESTful Web Clients<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhCzwB-uLxHgFpAVa0wIglBQAS_WfwnXeUuSQWfoTKtzH2PbQ1YuZH90EmN9StJaWPwQ75vv42vZPsZKtKJAREh4ebg4wj2TDwLuCBByUgIKj7b9Fzfrt7F-3SbHL0zo04kCO3zcGY_JQf1/s1600/book.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="499" data-original-width="381" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhCzwB-uLxHgFpAVa0wIglBQAS_WfwnXeUuSQWfoTKtzH2PbQ1YuZH90EmN9StJaWPwQ75vv42vZPsZKtKJAREh4ebg4wj2TDwLuCBByUgIKj7b9Fzfrt7F-3SbHL0zo04kCO3zcGY_JQf1/s200/book.jpg" width="152" /></a></div>
<a href="https://www.amazon.co.uk/RESTful-Web-Clients-Enabling-Hypermedia/dp/1491921900/ref=sr_1_1?ie=UTF8&qid=1507220030&sr=8-1&keywords=RESTful+Web+Clients">RESTful Web Clients</a> is written by guru <a href="https://www.infoq.com/profile/Mike-Amundsen">Mike Amundsen</a> who amongst other things co-authored <a href="https://www.amazon.co.uk/RESTful-Web-APIs-Leonard-Richardson/dp/1449358063/ref=sr_1_2?s=books&ie=UTF8&qid=1507228998&sr=1-2&keywords=RESTful+Web+APIs">RESTful Web APIs</a> with REST guru <a href="https://www.crummy.com/self/">Leonard Richardson</a> and <a href="http://intertwingly.net/blog/">Sam Ruby</a>.<br />
<br />
The book's primary focus is on the hypermedia aspect of REST, particularly from the client's perspective. As Roy Fielding detailed in this famous <a href="http://roy.gbiv.com/untangled/2008/rest-apis-must-be-hypertext-driven">blog post</a><i> "</i><i>if the engine of application state (and hence the API) is not being driven by hypertext, then it cannot be RESTful and cannot be a REST API</i><i>" </i>and let's face it, we have all seen APIs purporting to be REST with no hypermedia whatsoever with lots of coupling between client and server. Some of this is just down to basic <a href="http://www.dictionary.com/browse/ignorance">ignorance</a> and some of it probably down to misunderstanding the <a href="https://martinfowler.com/articles/richardsonMaturityModel.html">Richardson Maturity Model</a> <br />
<br />
Rather than begin with a summary of <a href="https://www.ics.uci.edu/~fielding/pubs/dissertation/fielding_dissertation.pdf">Fielding's dissertation</a> like most material on REST, this book begins with details of a simple web application that uses JSON RPC APIs. From the simple example Amundsen shows that while the JSON RPC approach functionally works, it results in a lot of coupling between client and server meaning that if the APIs need to change it will be difficult to do that easily as the client(s) with all its hardcoded of contracts will be impacted. <i>And we know software does need to change from time to time right?</i><br />
<br />
Amundsen distills the coupling with the JSON RPC approach into three distinct types which can be considered and assessed individually:<br />
<ul style="text-align: left;">
<li><b>Objects</b> - the JSON objects that appear in API responses. Clients need to be able to understand them to handle a simple a response to a GET request</li>
<li><b>Addresses</b> - the URLs clients needs to know to invoke requests</li>
<li><b>Actions</b> - details methods and arguments for all non-trivial operations. Again clients need to know this before invoking requests. </li>
</ul>
With the coupling clearly demonstrated, the scene is nicely set to move onto one of key advantages of a REST style archictecture: reducing coupling through hypermedia.<br />
<br />
To explain this advantage, Amundsen again uses the approach of specific examples. Firstly, by detailing the JSON hypermedia type <a href="http://stateless.co/hal_specification.html">HAL</a>. Using this approach reduces the Address coupling and examples of how generic response handling can be written on the client to leverage and take advantage of this decoupling are detailed. However HAL doesn't solve everything. Without a custom extension there is still coupling to the JSON Objects and the possible Actions available to the client. A work around to this is given and I would highly recommend anyone considering using HAL to read Chapter 4.<br />
<br />
Next up is another JSON hypermedia type known as <a href="http://hyperschema.org/mediatypes/siren">Siren</a>. <a href="https://github.com/kevinswiber">Kevin Swiber</a> designed Siren and registered it with <a href="https://www.iana.org/assignments/media-types/application/vnd.siren+json">IANA</a> in 2012. <br />
<span style="background-color: white; color: #333333; display: inline; float: none; font-family: serif; font-size: 18.88px; font-style: normal; font-weight: normal; letter-spacing: normal; text-align: left; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"></span> <br />
Siren splits response entities into:<br />
<ul style="text-align: left;">
<li>class - this is an array, the values of which indicate what the current resource represents e.g. <i>Customer, Person</i></li>
<li>properties - set of name-value pairs</li>
<li>entities - a list of <i>linked</i> and <i>representational</i> sub entities</li>
<li>actions - contains a set of valid operations of the associated entity and how to invoke those operations including a list of fields which match HTML5 input types (<i>hidden, text, number</i>). This is something <b>not</b> in HAL that helps reduce client-server coupling further</li>
<li>links - links to other resources. Each link has a class, href, rel, title, type property</li>
</ul>
<div>
</div>
<div>
Siren reduces coupling to Addresses and Actions, however it does not reduce coupling to Objects. There is <b>no</b> meta-data specification for the class type meaning the client has to hardcode the structure of the object somewhere. Like HAL it is possible to create a custom extension but this is not part of the Siren specification.<br />
</div>
<div>
The third hypermedia type detailed is <a href="http://amundsen.com/media-types/collection/format/">Collection+JSON</a> format (Cj). Interestingly, this format was designed by the author himself. The basic elements of a Cj message are:</div>
<div>
<ul style="text-align: left;">
<li>Links - Simlar to HAL and Siren links</li>
<li>Items - Similar to HAL and Siren properties and <i>also includes meta data about the properties</i></li>
<li>Queries - Information on how to construct various reads (HTTP GETs)</li>
<li>Templates - Information on how to construct various writes (HTTP POSTs, PUTs, DELETEs...)</li>
<li>error - information ref errors</li>
</ul>
</div>
<div>
The key point here is that since Cj includes the metadata about the items, it decouples the client from the Objects in the JSON responses something both HAL and Siren could only achieve with custom extensions.</div>
<div>
</div>
<div>
</div>
<div>
So which format? Well two good points to make here:</div>
<div>
<ol style="text-align: left;">
<li>That can be a practical decision and not just a technical one. You may prefer Cj because out of the box it achieves most decoupling, but your customer may be used to and prefer HAL.</li>
<li>Rather than trying to support every possible format, think about architecting so it possible to support extra formats if you need to. The approach suggested is described in the Amundsen's Representor pattern - which is inspired from the <a href="http://www.enterpriseintegrationpatterns.com/patterns/messaging/MessageTranslator.html"><i>Message Translator Pattern</i></a></li>
</ol>
So in summary, this is another great REST book from O'Reilly. The style of the book in general is pragmatic rather than academic. It really emphasizes and demostrates the importance of hypermedia in REST APIs and is backed up with practical examples. The central argument in the book is that Cj achieves the most decoupling. Even if it was written by the author, the argument is well made and I don't think it would be fair to make accusations of any selection bias since he does detail how you can extend Siren and HAL to achieve the same level of decoupling.<br />
<br />
Bottom line - if you want to understand the hypermedia aspects of REST, read this book. <br />
<br />
<br /></div>
</div>
Alex Staveleyhttp://www.blogger.com/profile/04378245339254378938noreply@blogger.com0tag:blogger.com,1999:blog-8965734277283465944.post-76642433452353075142017-08-13T13:10:00.000-07:002017-08-13T13:18:14.346-07:00From Developer to Architect: Patterns, Architecture Types, Soft Skills, and Continuous Delivery (Video Tutorial)<div dir="ltr" style="text-align: left;" trbidi="on">
<a href="https://www.oreilly.com/">O’Reilly</a> don’t just publish great technology books they also do some great video tutorials which are available from <a href="https://www.safaribooksonline.com/">Safari</a>. I recently just finished the series: <a href="https://www.safaribooksonline.com/tutorials/developer-to-architect-ots/">From Developer to Architect: Patterns, Architecture Types, Soft Skills, and Continuous Delivery</a> which consists of tutorial style videos about architectural patterns and anti-patterns, soft skills and a run through of some DevOps ideas and best practices. All the 17 tutorials are presented by two seasoned Architects <a href="http://www.wmrichards.com/">Mark Richards</a> and <a href="http://nealford.com/">Neil Ford</a> and take about <br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhf7QjBNr69UU34689RwFb9qGWWuaBQqOJQYU4E4zzb2f13jBXo7FFGp759At-MTI6BW7HXqGOTegu2JTZr70NMCNB2zg30j7yaWclZcNEjj5S2fucKDBmUcuWJJC7FHb24k6-3-spdvItO/s1600/NeilFord.jpeg" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="245" data-original-width="205" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhf7QjBNr69UU34689RwFb9qGWWuaBQqOJQYU4E4zzb2f13jBXo7FFGp759At-MTI6BW7HXqGOTegu2JTZr70NMCNB2zg30j7yaWclZcNEjj5S2fucKDBmUcuWJJC7FHb24k6-3-spdvItO/s200/NeilFord.jpeg" width="167" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Neil Ford</td></tr>
</tbody></table>
<br />
Even though the course is more at the fundamental / introductory level there are some topics and nuggets of information that are still useful either because you never had exposure to them (f<i>or example, not many projects use event driven architectures extensively</i>) or you have just forgotten them - in which case the course serves as an excellent refresher. <br />
<br />
Some highlights:<br />
<ol style="text-align: left;">
<li>The differences between Application, Integration and Enterprise Architecture are well detailed. </li>
<li>The <a href="https://martinfowler.com/bliki/ParallelChange.html">Expand and Contract pattern</a> is one mechanism to get over the DB coupling the <a href="http://microservices.io/patterns/data/shared-database.html">Shared Database</a> pattern introduces.</li>
<li>Shared database has a problem if the disparate applications all use their own caching. It becomes even more complex to determine when applications aren't looking at the latest data. </li>
<li>Even though ReST is generally stateless from the client's perspective, the resources have state and therefor using ETag is a good idea. Consideration should also be given to use <a href="https://httpstatuses.com/409">409</a> / <a href="https://httpstatuses.com/412">412</a> HTTP status codes when clients are using the wrong version of the Resource.</li>
<li>In a classical layered architecture, an open layer is one that can be by-passed e.g. Service Layer. However, having too many open layers completely defeats the purpose of a layered architecture.</li>
<li>Different patterns in Event Driven Architectures:</li>
<ul>
<li>Event processor / Mediator topology: Useful when ordering is required, achieved through orchestration</li>
<li>Broker topology: No central mediator, custom process components receive events directly</li>
</ul>
<li>Long running feature branches are anti-thetical to CI. <i>In my own humble opinion this is another reason why if you want to really to do CI use a tool that is good for branching / merging - GIT. </i></li>
<li><a href="https://medium.com/@jomoespe/dietzler-s-law-in-application-architecture-c4b2e0b99b0a">Dierzler's law</a> this is an excellent application architecture which reminds people not to be fooled by the initial illusions of progress in rapid prototyping. The user wants a full solution and doing a full solution is more difficult than the initial stages. I really believe anyone involved with rapid prototyping, hackathons or any form agile development should know this law inside out / upside down. </li>
<li>The <a href="https://docs.microsoft.com/en-us/azure/architecture/patterns/anti-corruption-layer">Anti Corruption layer</a> is a good pattern to use a facade / adapter approach to hide away bad legacy code.</li>
</ol>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhEy1PV_mHoTQW50q3q15NozFpaE1oZfABFuHNrebPsxDGam7gZ9NEkKxAeK7xJDCZsPdTRVa33T3PZqzORWRA39voPe-TxCmbqW8J12I47gRQsv1uTdqIOu1O1Wz5_BvvHWD8psgSmMAZP/s1600/mark_richards_headshot_bw_360-2.jpeg" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="340" data-original-width="360" height="188" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhEy1PV_mHoTQW50q3q15NozFpaE1oZfABFuHNrebPsxDGam7gZ9NEkKxAeK7xJDCZsPdTRVa33T3PZqzORWRA39voPe-TxCmbqW8J12I47gRQsv1uTdqIOu1O1Wz5_BvvHWD8psgSmMAZP/s200/mark_richards_headshot_bw_360-2.jpeg" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Mark Richards</td></tr>
</tbody></table>
A common theme throughout the course, is no matter what the architecture or technology you need to be prepared to change it and that means you need to be careful to get the level of coupling right. I would add to that an architecture that has a lot of tight coupling will struggle to deal with technical debt because if you need to fix one little part you impact everything. <br />
<br />
So any criticisms? Well not much. Perhaps some of the explanations are too abstract. For example, the Space architecture would be better explained with more specfics using a NoSql database to achieve the data splits. Other than that, there are large amounts dedicated to soft skills. While I don't doubt these to be important, you can get a lot of this by having a cup of coffee with a decent project manager or development manager and personally I prefer the technical stuff. Overall it's a great series for an into to software architecture and perhaps it would be nice to see a follow up with more deep diving. I would suspect that is exactly what the follow up series <a href="https://www.safaribooksonline.com/library/view/software-architecture-fundamentals/9781491901175/">Software Architecture Fundamentails Beyond the Basics</a> entails <br />
<br />
Lastly, although this tutorial series is primarily aimed at Developers who want to be (or have become) Architects, I think anyone involved with any sort of SDLC can benefit from it. Project, Dev Managers and Product Owners should all understand Dietzler's law for example. <br />
<br />
References<br />
<ol style="text-align: left;">
<li> <a href="https://www.safaribooksonline.com/library/view/software-architecture-patterns/9781491971437/copyright-page01.html">Software Architecture Patterns</a> </li>
<li> <a href="http://nealford.com/downloads/Software_Architecture_Fundamentals_Pt1_Neal_Ford_and_Mark_Richards.pdf">Software Architecture Fundamentals - Neil Ford PDF </a></li>
<li> <a href="https://www.safaribooksonline.com/library/view/software-architecture-fundamentals/9781491901175/">Software Architecture Fundamentals beyond the basics </a></li>
</ol>
</div>
Alex Staveleyhttp://www.blogger.com/profile/04378245339254378938noreply@blogger.com0tag:blogger.com,1999:blog-8965734277283465944.post-24297903751728685932017-07-16T04:28:00.000-07:002017-08-13T14:24:20.289-07:00Outputting the given, when, then, Extending Spock <div dir="ltr" style="text-align: left;" trbidi="on">
<b>Spock</b> is a Java testing framework, created in 2008 by Peter Niederwieser a software engineer with GradleWare, which facilitates amongst other things BDD. Leveraging this <a href="https://en.wikipedia.org/wiki/Behavior-driven_development">example</a>, a story may be defined as:
<br />
<pre><b>Story</b>: Returns go to stock
<b>As a</b> store owner
<b>In order to</b> keep track of stock
<b>I want to</b> add items back to stock when they're returned.
<b>Scenario 1:</b> Refunded items should be returned to stock
<b>Given</b> that a customer previously bought a black sweater from me
<b>And</b> I have three black sweaters in stock.
<b>When</b> he returns the black sweater for a refund
<b>Then</b> I should have four black sweaters in stock.
<b>Scenario 2:</b> Replaced items should be returned to stock
<b>Given</b> that a customer previously bought a blue garment from me
<b>And</b> I have two blue garments in stock
<b>And</b> three black garments in stock.
<b>When</b> he returns the blue garment for a replacement in black
<b>Then</b> I should have three blue garments in stock
<b>And</b> three black garments in stock.</pre>
Spock makes it possible to map tests very closely to scenario specifications using the same <i>given, when, then</i> <a href="https://martinfowler.com/bliki/GivenWhenThen.html">format</a>. In Spock we could implement the first scenario as:<br />
<pre class="brush: groovy; ruler: true; first-line: 1">class SampleSpec extends Specification{
def "Scenario 1: Refunded items should be returned to stock"() {
given: "that a customer previously bought a black sweater from me"
// ... code
and: "I have three black sweaters in stock."
// ... code
when: "he returns the black sweater for a refund"
// ... code
then: "I should have four black sweaters in stock."
// ... code
}
}
</pre>
<br />
What would be nice would be to ensure accurate mapping of test scenario requirements to test scenario implementation. We could get someway down this path if we could output the syntax of the <i>given, when, then</i> from our test. Spock allows us to add this functionality through its <a href="http://spockframework.org/spock/docs/1.0/extensions.html">extension framework</a>.<br />
<br />
So, let's say our BA is really curious and wants more confidence from the developer that they stuck to the same <i>given, when, then</i> format and their code is in-sync. They want to get this information easily. Developer could provide this information by first defining this annotation<br />
<pre class="brush: groovy; ruler: true; first-line: 1">import java.lang.annotation.*
import org.spockframework.runtime.extension.ExtensionAnnotation
@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.TYPE)
@ExtensionAnnotation(ReportExtension)
@interface LogScenarioDescription {}
</pre>
Followed by this implementation:
<br />
<pre class="brush: groovy; ruler: true; first-line: 1">import org.apache.log4j.Logger
import org.spockframework.runtime.AbstractRunListener
import org.spockframework.runtime.extension.AbstractAnnotationDrivenExtension
import org.spockframework.runtime.model.FeatureInfo
import org.spockframework.runtime.model.SpecInfo
class LogScenarioDescriptionExtension extends AbstractAnnotationDrivenExtension<logscenariodescription>; {
final static Logger logger = Logger.getLogger("scenarioLog." + ReportExtension.class);
@Override
void visitSpecAnnotation(Report annotation, SpecInfo spec) {
spec.addListener(new AbstractRunListener() {
@Override
void afterFeature(FeatureInfo feature) {
if (System.getEnv("logScenarios")) {
logger.info("***SCENARIO TEST:*** " + feature.name)
for (block in feature.blocks) {
logger.info(block.kind);
for (text in block.texts) {
logger.info(text)
}
}
}
}
})
}
}
</logscenariodescription></pre>
This will then be applied to the test
<br />
<pre class="brush: groovy; ruler: true; first-line: 1">@LogScenarioDescriptionExtension
class SampleSpec extends Specification{
//...
</pre>
When the test is executed it gives the following output:
<br />
<pre>***SCENARIO TEST:*** Scenario 1: Refunded items should be returned to stock
GIVEN
that a customer previously bought a black sweater from me
AND
I have three black sweaters in stock.
WHEN
he returns the black sweater for a refund
THEN
I should have four black sweaters in stock.
</pre>
output to a specific logfile for scenario logging by using the following log4j:
<br />
<pre>log4j.rootLogger=INFO, stdout
log4j.logger.scenarioLog.extension.custom=INFO, scenarioLog
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%m%n
log4j.appender.scenarioLog=org.apache.log4j.FileAppender
log4j.appender.scenarioLog.File=logs/scenario.log
log4j.appender.scenarioLog.layout=org.apache.log4j.PatternLayout
log4j.appender.scenarioLog.layout.ConversionPattern=%m%n
</pre>
and now you have a logfile that your BA, QA can read!
This helps foster an Agile culture of <a href="http://agilemanifesto.org/">collaboration</a> and <a href="https://www.agilealliance.org/glossary/atdd/">ATDD</a> where it possible to check that test scenarios implemented with those that were agreed.
</div>
Alex Staveleyhttp://www.blogger.com/profile/04378245339254378938noreply@blogger.com0tag:blogger.com,1999:blog-8965734277283465944.post-33905310596882235882016-06-08T09:51:00.004-07:002016-06-08T12:05:11.260-07:00Triggering a Client Cache Refresh<div dir="ltr" style="text-align: left;" trbidi="on">
More and more web sites are using a <a href="https://en.wikipedia.org/wiki/Single-page_application">single page</a> style architecture. This means there is a bunch of static resources (aka assets) including:<br />
<ul style="text-align: left;">
<li>JS </li>
<li>CSS</li>
<li>HTML templates</li>
<li>Images</li>
<li>...</li>
</ul>
all residing in the client's browser. For performance reasons, the static content is usually cached. And like all caches, eventually the data gets stale and the cache must be refreshed. One excellent technique for achieving this in the web tier is <i>Cache busting</i> (see: <a href="https://css-tricks.com/strategies-for-cache-busting-css/">here</a> and <a href="http://webassets.readthedocs.io/en/latest/expiring.html)">here</a>).<br />
However, even using this excellent pattern there still needs to be some sort of trigger to indicate that things have gone out of date. Some options:<br />
<h4 style="text-align: left;">
HTML5 Sockets</h4>
In this case a server can send a request to any client to say go and get the new stuff.<br />
<h4 style="text-align: left;">
Comet style polling</h4>
The Client consistently polls the server in the background asking if there is a new version available. When the server says there is, the client is triggered to start cache busting.<br />
<br />
Both of these are good approaches but in some edge cases may not be available:<br />
<ol style="text-align: left;">
<li>Architecture may not allow HTML5 sockets - might be some secure banking web site that just doesn't like it.</li>
<li>The <i>Comet</i> style polling might be too intensive and may just leave open edge cases where a critical update is needed but the polling thread is waiting to get fired. </li>
</ol>
<h4 style="text-align: left;">
Client refresh notification pattern</h4>
Recently, on a project neither of the above approaches were available, so I needed something else which I shall now detail. The solution was based on some existing concepts already in the architecture (which are useful for other things) and some new ones that needed to be introduced:<br />
<ol style="text-align: left;">
<li>The UI always knows the current version it is on. This was already burnt into the UI as part of the build process.</li>
<li>Every UI was already sending up the version it is on in a custom header. It was doing this for every request. <i>Note: this is very useful as it can make diagnosing problems easier. Someone seeing weird problems, nice to be able see their are on a old version straight away.</i></li>
</ol>
The following new concepts were then introduced:<br /><ol style="text-align: left;">
<li>The server would now always store the latest client version it has received. This is easy to do and again handy thing to have anyway.</li>
<li>The server would then always add a custom header to every response to indicate the latest client version it has received. Again, useful thing to have for anyone doing a bit of debugging with firebug or chrome web tools</li>
<li>Some simple logic would then be in the client, so that when it saw a different version in response to the one it sent up, it knows there's a later version out there and that's the trigger to start <i>cache busting</i>! This should be done in central place for example an Angular filter (<i>if you are using Angular</i>)</li>
<li>Then as part of every release, just hit the server with latest client in the smoke testing. </li>
<li>As soon as any other client makes a request, it will be told that there is a later client version out there and it should start cache busting.</li>
</ol>
<br />
So the clients effectively tell each other about the latest version but without ever talking to each other it's all via the server. It's like the <a href="https://sourcemaking.com/design_patterns/mediator">mediator</a> pattern. But it's still probably confusing. So let's take a look at a diagram.<br />
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgFoEIQ17h0ufnb6ztvXqX41hCzc3V01Wp6zN2OOKc1QpaToOzeF382KuDrmC1fFx5Fe2ldCSE3Nq2WUDKhuxu5NIdkrc5Bypbj8DJyUpTjgLvRt8I5piAhbjdPuUaONSvFEFsfOe8y0h5e/s1600/Updating+UI+Artifactes.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="215" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgFoEIQ17h0ufnb6ztvXqX41hCzc3V01Wp6zN2OOKc1QpaToOzeF382KuDrmC1fFx5Fe2ldCSE3Nq2WUDKhuxu5NIdkrc5Bypbj8DJyUpTjgLvRt8I5piAhbjdPuUaONSvFEFsfOe8y0h5e/s400/Updating+UI+Artifactes.png" width="400" /></a></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
With respect to diagram above: </div>
<div>
<ul style="text-align: left;">
<li>UI 1 has a later version (<i>for example 1.5.1</i>) of static assets than UI 2 (<i>for example 1.5.0</i>)</li>
<li>Server thinks the latest version static assets is 1.5.0</li>
<li>UI 1 then makes a request to the server and sends up the version of static assets it has e.g. 1.5.1</li>
<li>Server sees 1.5.1 is newer than 1.5.0 and then updates its latest client version variable to 1.5.1</li>
<li>UI 2 makes a request to the server and sends up the version of static assets it has which is 1.5.0</li>
<li>Server sends response to UI 2 with a response header saying that the latest client version is 1.5.1</li>
<li>UI 2 checks and sees that the response header client version is different to the version sent up and then starts busting the cache</li>
</ul>
<div>
Before the observant amongst you start saying this will never work in a real enterprise environment as you never have just one server (as in one JVM) you have several - true. But then you just store the latest client version in a distributed cache (e.g. <a href="http://infinispan.org/">Infinispan</a>) that they can all access. </div>
<div>
<br /></div>
<div>
Note: In this example I am using two web clients and one back end server. But note the exact same pattern could be used for back end micro-services that communicate with each other. Basically anything where there is a range of distributed clients (<i>they don't have to be web browsers</i>) and caching of static resources is required this could be used. </div>
</div>
<div>
<br /></div>
<div>
Until the next time, take care of yourselves. </div>
<div>
<br /></div>
<div>
<br /></div>
</div>
Alex Staveleyhttp://www.blogger.com/profile/04378245339254378938noreply@blogger.com1tag:blogger.com,1999:blog-8965734277283465944.post-55252623229322266612016-06-07T14:14:00.002-07:002016-06-08T12:15:25.577-07:00Why Agile can fail? <div dir="ltr" style="text-align: left;" trbidi="on">
Most Development teams now claim they are doing Agile and are usually doing some variant of Scrum. In this blog post I propose three reasons why teams can struggle with Scrum or any form Agile development. And conveniently they all begin with <b>R</b>.<br />
<h3 style="text-align: left;">
Requirements</h3>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjCgs8uTxCi_oGHOK-0farMv9VI4kIaq_DscAYCiSkyQ5ljVPnUDEA8Esn2CULdeeJKC6FBMSLny3g5lcAvMyytcDLFYOKPWMdqDO1Pj15sastkR_32C6hyphenhyphen8gYPDeRrrJQKmujPLD__eD1Z/s1600/ericsson-logo.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="44" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjCgs8uTxCi_oGHOK-0farMv9VI4kIaq_DscAYCiSkyQ5ljVPnUDEA8Esn2CULdeeJKC6FBMSLny3g5lcAvMyytcDLFYOKPWMdqDO1Pj15sastkR_32C6hyphenhyphen8gYPDeRrrJQKmujPLD__eD1Z/s200/ericsson-logo.png" width="200" /></a></div>
In the pre-agile days, requirements where usually very detailed and well thought out. For example, when I worked in Ericsson, before we moved to the Agile methodology RUP, the projects were based on a classical Waterfall model that was a very high standard (usually around CMM level 3). The requirements came from System experts who had lots of experience in the field, were very technically skilled and had incredible domain knowledge. Their full time job was to tease things out knowing that they had effectively only one chance to get it right. <br />
<br />
In the Agile world, with shorter iteration cycles and much more releases there are chances to make something right when you get it wrong. Functionality can be changed around much more easily. This is good. It is easier for customer collaboration and thus enable more opportunities to tweak, fine tune and get all stakeholders working together sculpting the best solution. <br />
<br />
However, because the penalty for getting requirements wrong is not as great as it is in the Waterfall model it can mean that the level of detail and clarity in requirements can start becoming insufficient and before you know development time in the sprint gets wasted trying to figure what is actually required. The key is to get the balance right. Enough detail so that there can be clear agreement across the dev team, customer and any other stakeholders about what is coming next, but not so much detail that people are just getting bogged down in paralysis analysis and forgetting that they are supposed to be shipping regularly.<br />
<br />
I suggest one process for helping get the balance between speed and detail for your requirements in this blog <a href="http://dublintech.blogspot.ie/2016/05/requirements-in-agile-world.html">post</a>.<br />
<h3 style="text-align: left;">
Releases</h3>
In the Agile methodology Scrum you are supposed to release at the end of every sprint. This means instead of doing 1 - 4 releases a year you will be doing closer to 20 <i>if not more</i>. If your release is painful your team will struggle with any form of Agile. For example, say a full regression test, QA, bug fixing, build, deploy etc takes 10 days (including fixing any bugs found during smoke testing) it means that 20 * 10 = 200 man days are going to releases. Whereas in the old world, with 4 release it would just be 4 * 10 = 40 days. In case it's not obvious, that's a little bit regressive.<br />
<br />
Now, the simple maths tells us that a team with long release cycle (for whatever reason) will struggle releasing regularly and will thus struggle with any Agile methodology.<br />
<br />
To mitigate this risk happening, it is vital that the team has superb CI with very high code coverage and works with a very strict CI culture. This includes:<br />
<ul style="text-align: left;">
<li>Ensuring developers are running full regressing tests on feature branches before merging into dev branches to minimise broken builds on dev branches</li>
<li>Fix any broken build as a priority</li>
<li>No checking into a broken build</li>
<li>Tests are written to a high quality - they need to be as maintainable as your source code</li>
<li>Coding culture where the code is written in a style so it is easily testable (I'll cover this in a separate blog post)</li>
<li>CI needs to run fast. No point having 10,000 tests, if they take 18 hours to run. Run tests in parallel if you have to. If your project is so massive that it really does take 18 hours to run automated tests, you need to consider some decomposition. For example, a micro-service architecture where components are in smaller and more manageable parts that can be individually released and deployed in a lot less than 18 hours.</li>
</ul>
For more information on how to achieve great CI, see <a href="http://martinfowler.com/articles/continuousIntegration.html">here</a> and <a href="http://blogs.collab.net/devopsci/ten-best-practices-for-continuous-integration#.V1cxqHUrKK4">here</a>.<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgqvzq1fNSjgtGA0XYc3IMVGthZXz35ZEhln6Lt-p-fe-YGHk-ORWcQ9p-SUp5G1SnoKvtuaHEVEuVbbEnmkLU-W61gi8QH-jzMZxJCcvaM7XsbtBCA39b-wgg8bzya0D2t3S6nH7w9WXmK/s1600/newrelease.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgqvzq1fNSjgtGA0XYc3IMVGthZXz35ZEhln6Lt-p-fe-YGHk-ORWcQ9p-SUp5G1SnoKvtuaHEVEuVbbEnmkLU-W61gi8QH-jzMZxJCcvaM7XsbtBCA39b-wgg8bzya0D2t3S6nH7w9WXmK/s200/newrelease.jpg" width="200" /></a>By mastering automated testing, the release cycle will be greatly shortened. The dev team should be aiming towards a <a href="https://www.thoughtworks.com/continuous-delivery">continuos delivery model</a> where in theory any check could be released if the CI says it is green. Now, this all sounds simple, but it is not. In practise you need skilled developers to write good code and good tests. But the better you are at it, the easier you will be able to release, the easier you will be able to truly agile.<br />
<br />
<i>Note: One of the reasons why the micro-services architectural style has become popular is because it offers an opportunity to avoid big bang releases and instead only release what needs to be. That's true. However, most projects are not big enough or complex enough to need this approach. Don't just jump on a bandwagon, justify every major decision.</i><br />
<h3 style="text-align: left;">
Roles </h3>
The most popular agile methodology Scrum only defines 3 Roles:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEisGqmkM0CovpUBdnVtOaM-iJsOOYpOOX52wx1MZsNa6eNbwDMTLnhyphenhyphenLOJKptuDwaF8nR9T2SStT2BMpN020fZb-3QV_UY2iZ06zal4HnxqyDB1cju8bN2kwyFU_fM-Tk_LRPRJjNnfm0LG/s1600/scrummaster-retrospective-management-agile.jpeg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEisGqmkM0CovpUBdnVtOaM-iJsOOYpOOX52wx1MZsNa6eNbwDMTLnhyphenhyphenLOJKptuDwaF8nR9T2SStT2BMpN020fZb-3QV_UY2iZ06zal4HnxqyDB1cju8bN2kwyFU_fM-Tk_LRPRJjNnfm0LG/s200/scrummaster-retrospective-management-agile.jpeg" width="161" /></a></div>
<ul style="text-align: left;">
<li>Product Owner </li>
<li>Scrum Master</li>
<li>Dev Team. </li>
</ul>
That's it. But wait sec! No Tech Lead, no Architect, no QA, no Dev manager - surely you need some of these on a project. <br />
<i>Of course you do</i>. But this can be often forgotten. Scrum is a mechanism to help manage a project, it is not a mechanism to drive quality<br />
engineering, quality architecture and minimise technical debt. Using Scrum is only one aspect of your process. While your Scrum Master might governs process they don't have to govern architecture or engineering. They may not have a clue about such matters. If all the techies are just doing their own stories trying to complete them before the next <i>show and tell </i>and no-one is looking at the big picture, the project will quickly turn into an unmanageable ball of spaghetti.<br />
<br />
This is a classical mistake at the beginning of a project. Because at the beginning there is no Tech debt. There are also no features and no bugs and of course all of this is because there is no code! But, the key point is that <i>there is no tech debt.</i> Everyone starts firing away at stories and there is an illusion of rapid progress but if no-one is looking at the overall big picture, the architecture, the tech debt, the application of patterns or lack of, after a few happy sprints the software entropy will very quickly explode. Meaning that all that super high productivity in the first few weeks of the progress will quickly disappear.<br />
<br />
To mitigate this, I think someone technical has to back away from the coding (<i>especially the critical path</i>) and focus on the architecture, the technical leading, enforcing code quality. This will help ensure good design and architecture decisions are made and non-functional targets of the system are not just well defined but are met. It is not always a glamorous job but if it ain't done, complexity will creep in and soon make everything from simple bug fixing, giving estimates, delivering new features all much harder than they should be.<br />
<br />
In the Scrum world it is a fallacy to think every technical person must be doing a story and nothing else. It is the equivalent of saying that everyone building a house has to be laying a brick and then wondering why the house is never finished because when the bricks never seem to line up.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgD1Ellu6odTQwXaRiT83Ad6I3uPBQf6D-pT9DCVC1UFf2nB_BOqdbPlfPUgcjhIrUNyG5Ww63ekbAuyhexlYEgCgap6Ld5wbYXwy1Wu-3e3FsWH10lF0sItDJwLq8N7viVARFo3mzUCC7W/s1600/IMG_0735_Home_600_DJ.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="166" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgD1Ellu6odTQwXaRiT83Ad6I3uPBQf6D-pT9DCVC1UFf2nB_BOqdbPlfPUgcjhIrUNyG5Ww63ekbAuyhexlYEgCgap6Ld5wbYXwy1Wu-3e3FsWH10lF0sItDJwLq8N7viVARFo3mzUCC7W/s320/IMG_0735_Home_600_DJ.jpg" width="320" /></a></div>
<br />
<br />
Someone has to stand back ensure that people's individual work is all coming together, the tech debt is kept at acceptable levels and any technical risks are quickly mitigated.<br />
<br />
Until the next time, take care of yourselves.<br />
<br />
<br />
<br />
<br /></div>
Alex Staveleyhttp://www.blogger.com/profile/04378245339254378938noreply@blogger.com2tag:blogger.com,1999:blog-8965734277283465944.post-43477297554992993142016-06-02T13:12:00.001-07:002016-06-02T15:38:45.607-07:00Agile Databases<div dir="ltr" style="text-align: left;" trbidi="on">
Any project following an Agile methodology will usually find itself releasing to production
at least 15 - 20 times per year. Even if only half of these releases involve database changes, that's 10 changes to production databases so you need a good lean process to ensure you get a good paper trail but at the same time you don't want something that that will slow you just unduly down. So, some tips in this regard:<br />
<h3 style="text-align: left;">
Tip 1: Introduce a DB Log table</h3>
<div style="text-align: left;">
Use a DB Log table to capture every script run, who ran it, when it was run, what ticket
it was associated with etc.
Here is an example DDL for such a table for PostGres:</div>
<pre class="brush: text; ruler: true; first-line: 1">create sequence db_log_id_seq;
create table db_log (id int8 not null DEFAULT nextval('db_log_id_seq'), created timestamp not null, db_owner varchar(255), db_user varchar(255), project_version varchar(255), script_link varchar(255), jira varchar(255));
</pre>
W.R.T. the table columns:<br />
<ul style="text-align: left;">
<li>id - primary key for table. </li>
<li>timestamp - the time the script was run. This is useful. Believe me. </li>
<li>db_owner - the user who executed the script. </li>
<li>db_user - the user who wrote the script </li>
<li>project_version_number - the version of your application / project the script was generated in.</li>
<li>scrip_link - a URL link to a source controlled version of the script </li>
<li>jira - a URL to the ticket associated with the script. </li>
</ul>
<h3 style="text-align: left;">
Tip 2: All Scripts should be Transactional</h3>
<div>
For every script, make sure it happens within a transaction and within the transaction make sure there is an appropriate entry into the db log table.
For example, here is a script which removes a column<br />
<pre class="brush: text; ruler: true; first-line: 1">BEGIN;
ALTER TABLE security.platform_session DROP COLUMN IF EXISTS ttl;
INSERT INTO db_log (
db_owner, db_user, project_version, script_link, jira, created)
VALUES (
current_user,
'alexstaveley',
'1.1.4',
'http://ldntools/labs/cp/blob/master/platform/scripts/db/updates/1.1.4/CP-643.sql',
'CP-643',
current_timestamp
);
COMMIT;</pre>
<h3 style="text-align: left;">
Tip 3: Scripts should be Idempotent</h3>
Try to make the scripts idempotent. If you have 10 developers on a team, every now and again someone will run a script
twice by accident. Your db_log will tell you this, but try to ensure that when accidents happen that there is no serious damage. This means you get a simple <i>fail safe,</i> rather than some newbie freaking out. In the above script, if it is run twice the answer will be the exact same.<br />
<h3 style="text-align: left;">
Tip 4: Source Control your Schema</h3>
Source control a master DDL for the entire project. This is updated anytime the schema changes.
Meaning you have update scripts <i>and</i> a complete master script containing the DDL for entire project.
The master script is run at the beginning of every CI, meaning that:<br />
<ul style="text-align: left;">
<li>Your CI always starts with a clean database </li>
<li>If a developer forgets to upgrade the master script, the CI will fail and your team will quickly know the master script needs to be updated.</li>
<li>When you have a master script it gives you two clear advantages: </li>
<ul>
<li>New developers get up and running with a clean database very quickly</li>
<li>It becomes very easy to provision new environments. Just run the master script! </li>
</ul>
</ul>
<h3 style="text-align: left;">
Tip 5: Be Dev Friendly</h3>
Make it easy for developers to generate the master script. Otherwise when the heat is on, it won't get done.<br />
<h3 style="text-align: left;">
Tip 6: Upgrade and Revert</h3>
For every upgrade script write a corresponding revert script.
Something unexpected happens in production, you gotta be able to reverse the truck back out!<br />
<pre class="brush: text; ruler: true; first-line: 1">BEGIN;
ALTER TABLE security.platform_session ADD COLUMN hard_ttl INT4;
UPDATE security.platform_session SET hard_ttl = -1 WHERE hard_ttl IS NULL;
ALTER TABLE security.platform_session ALTER COLUMN hard_ttl SET NOT NULL;
ALTER TABLE security.platform_session ADD COLUMN ttl INT4;
UPDATE security.platform_session SET ttl = -1 WHERE ttl IS NULL;
ALTER TABLE security.platform_session ALTER COLUMN ttl SET NOT NULL;
INSERT INTO db_log (
db_owner, db_user, platform_version, script_link, jira, created)
values (
current_user,
'alexstaveley',
'1.1.4',
'http://ldntools/labs/cp/blob/master/platform/scripts/db/reverts/1.1.4/revert-CP-463.sql',
'CP-463',
current_timestamp
);
COMMIT;
</pre>
<br />
Until the next time take care of yourselves.<br />
<br /></div>
</div>
Alex Staveleyhttp://www.blogger.com/profile/04378245339254378938noreply@blogger.com0tag:blogger.com,1999:blog-8965734277283465944.post-88589015218740194562016-05-19T14:27:00.004-07:002020-04-25T10:13:00.827-07:00Immutable pointers - a pattern for modular design to help achieve microservices<div dir="ltr" style="text-align: left;" trbidi="on">
<div>
Modular design has a lot of benefits, including:</div>
<div>
<ul style="text-align: left;">
<li>making it easier to predict the impacts from a change</li>
<li>helping developers work in parallel</li>
</ul>
</div>
<div>
But, it is <i>much much much</i> harder than people think. For a start, the abstractions have to be at the right level. Too high and they can become meaningless and too low and they cease to be abstractions, as they will end up having way too many dependencies (<a href="http://codebetter.com/raymondlewallen/2005/07/15/coupling-abstractness-stability-measuring-and-applying-code-metrics/">efferent</a> and <a href="http://codebetter.com/raymondlewallen/2005/07/15/coupling-abstractness-stability-measuring-and-applying-code-metrics/">afferent</a>)</div>
<div>
<br /></div>
<div>
In a recent green field project, which was in the area of digital commerce, I was intent on achieving good modular design in the back end for the reasons outlined above and because the project had potential to grow into a platform (<i>more than likely microservices</i>) that would be used by multiple teams. To achieve the modular design, after some white boarding the back end was split up into a bunch of conceptual components to achieve shopping functionality. The core components are listed below.</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhLsvwgM9BKKybqcYyrVCiNBJvxEY-rY6tAObYvXehDI0C8iTvrcJEVMyLEpBoUVcv2F_Bhq3A713noY18QgxLoTZYeZS8xwbG7-u4CHQcBHadBHs3XYq2URwy22GhjW1-QzIrBivBpd7ol/s1600/modularcomponents.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="120" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhLsvwgM9BKKybqcYyrVCiNBJvxEY-rY6tAObYvXehDI0C8iTvrcJEVMyLEpBoUVcv2F_Bhq3A713noY18QgxLoTZYeZS8xwbG7-u4CHQcBHadBHs3XYq2URwy22GhjW1-QzIrBivBpd7ol/s320/modularcomponents.png" width="320" /></a></div>
<div>
<br /></div>
<div>
<div>
<ul style="text-align: left;">
<li>Shopping component - Core shopping functionality e.g. Shopping Cart management</li>
<li>User component - user management, names, addresses etc</li>
<li>Purchases - details about previous transactions</li>
<li>Merchant - Gateway to merchant API to get information about merchant products etc. </li>
<li>Favourites - Similar to an Amazon wishlist</li>
</ul>
</div>
</div>
<div>
Now, every software project uses the word "<i>component"</i> differently. In this project, a <i>component</i> was strictly defined as something that did something useful and contained:</div>
<div>
<ul style="text-align: left;">
<li>domain classes </li>
<li>services</li>
<li>a database schema (could be its own database, but the idea was to isolate it from any other components persistence) </li>
<li>its own configuration</li>
<li>its own dedicated tests</li>
<li>its own exception codes domain </li>
</ul>
<div>
The outside world could access a component via a bunch of ReSTful endpoints. </div>
</div>
<div>
<br /></div>
<div>
Any component could be individually packaged, deployed etc. Now, the astute out there will </div>
<div>
be thinking, "<i>this sounds like microservices</i>", well they almost were. For this project, </div>
<div>
some of them were co-located but they were architected so that deploying them out into </div>
<div>
individual deployed artefacts (and hence a microservices approach would be easy). </div>
<div>
<br /></div>
<div>
Ok, to reiterate, the goal was achieve a very clean modular design. This meant, that I didn't want </div>
<div>
any dependencies from one component's database scheme to another and for this blog post we are only going to focus on how that aspect of the modularity was achieved. </div>
<div>
<br /></div>
<div>
Now, looking at the above components, it doesn't take long to see that isn't going to be so easy. For example: </div>
<div>
<ul style="text-align: left;">
<li>A shopping cart (in the <i>Shopping component</i>) will have a reference to a User (in the <i>User component</i>)</li>
<li>A shopping cart item (<i>Shopping component</i>) will have a reference to a Product (<i>Merchant component</i>)</li>
<li>A Shopping cart (Shopping component) will have a reference to a shipping Address (<i>User component</i>) </li>
</ul>
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhcwTiAAW7YgqjHlMAwgOdO3I8ui70O9nEnyzSgoybeaNEE6uRd3pZ1Y247htratigPg8umGbpIujqeZ8ZTAs0HXb_UgifnC9DjMCfibgsGpkOzECodb9EPOwye0g7Hgi_G_85w7njWCDit/s1600/domains.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="248" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhcwTiAAW7YgqjHlMAwgOdO3I8ui70O9nEnyzSgoybeaNEE6uRd3pZ1Y247htratigPg8umGbpIujqeZ8ZTAs0HXb_UgifnC9DjMCfibgsGpkOzECodb9EPOwye0g7Hgi_G_85w7njWCDit/s320/domains.png" width="320" /></a></div>
<div>
<br /></div>
<div>
So the challenge of achieving modularity in the persistence tier should now be becoming more clear. References of some sort across components need to be persisted. Immediately, any developer will ask, </div>
<div>
"<i>Wait a sec, if we just use foreign keys for these inter schema references we get ACID and referential </i></div>
<div>
<i>integrity for free!</i>" True. But then you are loosing modularity and introducing coupling. Say you want to move your products (in the <i>Merchant Component</i>) away from a relational database and use something like Elastic or Mongo DB instead - to leverage their searching capabilities. That foreign key isn't so useful now is it? </div>
<div>
<br /></div>
<div>
Ok, so first of all in looking for a solution here, I thought about all the references that were across </div>
<div>
components to see if there was anything in common with them. One thing that was obvious was that </div>
<div>
they were generally all <i>immutable</i> in nature. For example:</div>
<div>
<ul style="text-align: left;">
<li>When a Cart Item (Shopping component) points to a Product (Merchant component) it points to that product only. It never changes to point to another product. </li>
<li>When a Shopping Cart (Shopping component) points to a User (User component), it is also immutable. <i>My shopping cart is always mine, it never changes to be someone else's.</i></li>
</ul>
</div>
<div>
So I was now starting to think about preferences:</div>
<div>
<ol style="text-align: left;">
<li>Avoid cross component dependencies if you can (this should be kinda obvious)</li>
<li>If you have to have them, strive for immutable references.</li>
</ol>
</div>
<div>
So, next up was to have a name for this type of relationship - which I was calling "Immutable pointer" in design and architecture documents. But, for actual code I needed something more succint. The database schema was already using "id" for primary keys and "{name_other_relationship}_id" for foreign keys. So I decided all cross component relationships were named as the the name of the entity being pointed to and "ref". </div>
<div>
<br /></div>
<div>
So, some concrete examples:</div>
<div>
<ul style="text-align: left;">
<li>userRef (ShoppingCart pointing to the user)</li>
<li>productRef (CartItem pointing to the product)</li>
<li>shippingAddressRef (ShoppingCart pointing to the ShippingAddress)</li>
</ul>
</div>
<div>
This meant anytime anyone saw something like "xyzRef" in code, schema or logfiles they knew it was a cross component reference. In case it wasn't obvious, Ref was an abbreviation for <i>Reference</i>.</div>
<div>
<br /></div>
<div>
Next up was to decide on the format for the actual Refs. I took a bit of inspirational from that thing call the internet, which of course has similar concepts where abstractions: <i>web sites of web pages </i>contain immutable pointers to other abstractions: <i>hyperlinks to web pages in other web sites</i>.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgBNAvrnRct5y0_Wz5CGcHSZF7RiKX4jl1xSFH8wuy63cvi9c_S_N1gVpd6z5UIs9nML1-purbw4J1C85abBhIEk2pWHcR2GP0DC-juO3CS8ABdax1sBYW4xFx9MswI9t5lv03abCAGiqZM/s1600/hyperink.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="135" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgBNAvrnRct5y0_Wz5CGcHSZF7RiKX4jl1xSFH8wuy63cvi9c_S_N1gVpd6z5UIs9nML1-purbw4J1C85abBhIEk2pWHcR2GP0DC-juO3CS8ABdax1sBYW4xFx9MswI9t5lv03abCAGiqZM/s200/hyperink.jpg" width="200" /></a></div>
<br /></div>
<div>
<br /></div>
<div>
So similar to hyperlinks and URLs, the refs would follow a hierarchical name spacing format.</div>
<div>
Some good team input from senior technical member suggested to continue the inspiration from the Web and start the hierarchical names with cp://. CP for <i>Commerce Platform</i> the name of the project. <i>This was analogous to http:// </i> I thought this was good idea as it indicates that our platform generated the reference. Again, this meant they stood out in logfiles etc and could be differentiated from any downstream components also using hierarchical type references but in a different context. </div>
<div>
<br /></div>
<div>
The key point of the ref was that when generated it should of course be unique. To achieve this </div>
<div>
a mixture of database primary keys or something unique about the data (e.g. product skus were used)</div>
<div>
<br /></div>
<div>
So some examples: </div>
<div>
<ul style="text-align: left;">
<li>userRef -> cp://user/{uuid of user}</li>
<li>productRef -> cp://merchant/{uuid of merchant}/product/{sku of product}</li>
<li>cardRef -> cp://user/{uuid of user}/card/{uuid of card}</li>
</ul>
</div>
<h4 style="text-align: left;">
Always immutable?</h4>
<div>
As stated, the first preference was to avoid the cross component reference. The second preference</div>
<div>
was to use the immutable pointer (ref pattern). However, what about an edge case where the cross component reference could be mutable. Could this happen? Well it could. Easiest to explain by an example. </div>
<div>
<br /></div>
<div>
Every shopping cart doesn't just have a User, it also has a selected shipping address where the contents purchased will be shipped to. In the domain model, the user's address lived in the User component. But unlike the other cross component references the shipping address could change. Consider your Amazon shopping cart. Imagine you are on the checkout screen with your selected card and your selected address, but before you proceed to checkout you go into your user preferences and delete your card and address. This project had to facilitate similar scenarios. So the User deletes the address that a shopping card id pointing to what should happen?</div>
<div>
<br /></div>
<div>
So for inspiration for solution here, we can look to the various NoSQL patterns and in particular </div>
<div>
one of the most popular is eventually consistency. What this says is that unlike <a href="https://en.wikipedia.org/wiki/ACID">ACID</a> you don't </div>
<div>
always need consistency straight away, all the time. In certain cases it is okay to allow inconsistency</div>
<div>
on the basis that the system is able to reconcile itself. </div>
<div>
<br /></div>
<div>
So in this case:</div>
<div>
<ol style="text-align: left;">
<li>The shopping cart is pointing to a specific shopping address using an addressRef. </li>
<li>The user deletes that address by hitting a ReST endpoint in the user component.</li>
<li>This means the shopping cart will point to an address that doesn't exist. The system is inconsistent.</li>
<li>The next time the user reads the shopping cart, in the request handling the shopping component asks the user component if the address with this address ref sill exists and if it doesn't removes the pointer. </li>
<li>This system is now consistent</li>
</ol>
</div>
<div>
So with the architect hat on it is really important we get this all right. Otherwise the goal </div>
<div>
of modular design falls apart. </div>
<div>
<br /></div>
<div>
In this case, it is worth reiterating the strategy <a href="https://www.youtube.com/watch?v=FGBhQbmPwH8">one more time</a>:</div>
<div>
<ol style="text-align: left;">
<li>Avoid cross component references. Doesn't matter how great your pattern is, if you have a lot of cross component references, it is more than likely you have got the component abstractions at too low a level.</li>
<li>Favour immutability. In general immutability means less code paths, less edge cases, less complexity in code. </li>
<li>Eventual consistency. </li>
</ol>
</div>
<div>
Software Architecture is about trade offs, and finding the right balance. In this case, it was the balance between clean modular design but not going over board with it so that it become impossible to achieve. </div>
<div>
<br /></div>
<div>
For anyone trying to do microservices, I would strongly recommend trying to master how you would do modular design in your architecture first. If you can't do this, when you add in the complexity of the network things get very complicated. </div>
<div>
<br /></div>
<div>
</div>
<div>
<br /></div>
</div>
Alex Staveleyhttp://www.blogger.com/profile/04378245339254378938noreply@blogger.com0tag:blogger.com,1999:blog-8965734277283465944.post-62850857293692052282016-05-09T13:04:00.002-07:002016-05-12T13:33:21.649-07:00Requirements in an Agile World <div dir="ltr" style="text-align: left;" trbidi="on">
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjXyovMpyPtu7_pxz2-ynIy3W9NeughXOi1y1SwWUHTDiQoPAU_kZkZAho9FYAQnj8H-sbhnIb2Sexp2WkTwn5OXibiPQJoPm7IuJgmVDPV-voADccXI-A_MhA7NRuAZ90u4YNbZ9Fm_xmK/s1600/thomas-marten-quote-it-is-now-clear-that-it-is-unclear.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjXyovMpyPtu7_pxz2-ynIy3W9NeughXOi1y1SwWUHTDiQoPAU_kZkZAho9FYAQnj8H-sbhnIb2Sexp2WkTwn5OXibiPQJoPm7IuJgmVDPV-voADccXI-A_MhA7NRuAZ90u4YNbZ9Fm_xmK/s200/thomas-marten-quote-it-is-now-clear-that-it-is-unclear.jpg" width="200" /></a>On a recent project, Stories with unclear requirements were ending up with developers. It was a pain point. Time was spent trying to clarify missing pieces meaning less time to write great code and meaning developers ended up under even more pressure to get things done.<br />
<br />
Now, there are many reasons why Development teams may struggle to adopt to a well structured agile process (whether it be Scrum, Kanban, XP or whatever). I think one reason is when requirements start going into Sprints that aren't detailed enough. They lack sufficient specification. This, of course, then creates further problems, including: bad estimations, drop in engineering quality and the correct functionality just not getting delivered. If this sounds in any way at all familiar, please read on.<br />
<br />
In our case, we did have a development process which is best summarised by the Sprint board swim-lanes we were using:<br />
<ul style="text-align: left;">
<li><b>Ready for developmen</b>t - Story was in sprint but no-one had started it. </li>
<li><b>In Progress</b> - developer working on story </li>
<li><b>Code Review </b>- 2 people usually code reviewing. We were good at this. </li>
<li><b>UX review</b> - UX or Product person would look at output to confirm everyone on same page.</li>
<li><b>Done</b> - Story merged, finished testing and ready to be released. </li>
</ul>
The <i>unclear requirements </i>problem was happening because the process didn't vet stories sufficiently
before they were passed over to Developers. I decided to attempt to fix this by introducing a new process for the
team specifically for capturing requirements. <br />
<br />
Some goals I had for this process:<br />
<ul style="text-align: left;">
<li>It should be JIRA centric. Developers, analysts, product owners, leads,
should be able to get everything they needed from the one tool which in our case was JIRA. Time to cut down on Excel, Word, random emails, meeting notes on someone's laptop as much as possible and put all the important information in one shared place. It doesn't necessarily have to be JIRA, could be Asana, Trello or any Agile friendly
project management tool that your team are using.</li>
<li>The stories had to be vetted. Strict criteria had to be met before stories could be handed
over to Developers. The vetting needed to be done by key technical people. It was fine someone with a non-technical
background bringing forward an idea - they are usually the people to do this as they are closer to the customers. But whatever the idea it had to be technically feasible. And if it needed to be tweaked to make it
so, it was better that happened as early as possible. </li>
<li>Key stakeholders had to come in at key stages in requirements capturing. For example, UX expertise for UX design. </li>
<li>Allow people work in an asynchronous manner. The reality was story evolution required
different key people at different stages. The process needed to facilitate this. </li>
</ul>
<h3 style="text-align: left;">
In the beginning...</h3>
So, not being a JIRA expert, I first created a JIRA project that I could play around without impacting the existing project.
I then set about attempting to create a new workflow for the requirements capturing process.<br />
<h3 style="text-align: left;">
JIRA statuses</h3>
Ok so every workflow involves moving an issue through various statuses. I had a look at the existing JIRA issue statuses (that came with JIRA out of the box) both for inspiration and because I wanted to use
existing industry norms as much as possible.
So after having a quick look, I came up with some statuses I thought made sense:<br />
<ul style="text-align: left;">
<li><b>Idea</b> - this would represent just a brain dump of someone's idea. Could be just a sentence, a paragraph or heck even just a picture! </li>
<li><b>Story Definition</b> - at this state, the expectation would be that the acceptance criteria and the usual story stuff would be there. </li>
<li><b>UX Design </b>- screen shots, wire frames etc would be added by someone with UX expertise which would guide the Developer
on how things should look. </li>
<li><b>Technical Review</b> - this step would be when the serious vetting was done. If there was not enough information Technical Review would fail and the issue would be sent back to Story Definition state for further work. Two other important functions of this step:</li>
<ol>
<li>It provided an opportunity to provide architectural advice if it was needed </li>
<li>It prompted for an initial estimate for the work. Idea was to really capture if this was a small, medium or large piece of work. The estimate could always be refined.</li>
</ol>
<li><b>Ready for development</b> - this step would indicate that the technical review had passed and the story was ready to go into a sprint
and be taken by a Developer.</li>
</ul>
<h4 style="text-align: left;">
<span style="font-weight: normal;">The observant amongst you will note that the last step in the requirements capturing process is the first in the developer process. This was deliberate to emphasis the separation of flows and the sense of continuity when the issue ended up in a sprint with a Developer.</span></h4>
<h4 style="text-align: left;">
Workflow </h4>
<div>
So next up was to put all those states into some sort of workflow. This is best explained by the JIRA workflow diagram below. </div>
<div>
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhpHucm10tf2jDX0BMGNaBqv7McP6vONpMqPf6PQOX1Xgf5Rw_ddGV8yv9vM5S8D_W9_Rz6EGRqVpK3OuWf9Fg70QL8nbtgstZGIog4wKE-JfYfbWKFiI-Ek8mwlLhiUxbcUaPfW1VGDpr4/s1600/Screen+Shot+2016-05-08+at+22.25.27.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="286" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhpHucm10tf2jDX0BMGNaBqv7McP6vONpMqPf6PQOX1Xgf5Rw_ddGV8yv9vM5S8D_W9_Rz6EGRqVpK3OuWf9Fg70QL8nbtgstZGIog4wKE-JfYfbWKFiI-Ek8mwlLhiUxbcUaPfW1VGDpr4/s640/Screen+Shot+2016-05-08+at+22.25.27.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Requirements workflow</td></tr>
</tbody></table>
<div>
With respect to the above diagram, the following are the key points: </div>
<div>
<ol style="text-align: left;">
<li>The general direction is: Idea -> Story Definition -> UX design -> Technical Review -> Ready for Development. </li>
<li>It is possible for the issue to go forwards and backwards between several states. For example, it is possible to be in Technical Review and then go back to Story Definition, back to Technical Review and then back to Story Definition. The idea here is not only have some vetting before a story goes to Ready for development,
but so that an analyst could gather some technical questions regarding feasibility and get them answered to help them evolve their
story definition. There might be an element of to and fro between key stakeholder and the workflow captured this. It it worth re-emphasising that the Technical Review and Story Definition stages are usually handled by two different people; a technical expert for the former and a product expert for the latter. The process must help them work together. </li>
<li>I added an On Hold and a Closed stage. On Hold was for stories that just became less important but not irrelevant.
The irrelevant were just Closed. The advantage of the Closed stage as opposed to just deleting them was that you had a record of the
work that was done and the reason why you didn't proceed with the original idea. </li>
</ol>
<h3 style="text-align: left;">
New fields</h3>
</div>
<div>
Next up was to add more fields. Some of these came from feedback from the team. This was good. The more people who contributed to the process the
more buy in, the more likely the new process was going to work. </div>
<div>
<ul style="text-align: left;">
<li><b>Business Driver</b> - idea here was to capture why we were doing this JIRA? Was it an internal idea, coming from a customer or what?
Important that everyone knew this, Analyst, Developer, Lead etc. </li>
<li><b>Acceptance criteria </b>- a logical place to formally specify the acceptance criteria. </li>
<li><b>Technical review</b> - a logical place to detail the technical review. This would include architectural advise on how the Developer
should approach the story. </li>
</ul>
<h3 style="text-align: left;">
Checkpoints</h3>
</div>
<div>
As stated, the plan for this process was to allow stories evolve from a simple idea into something that was formally agreed, that could just be implemented by a Developer. To facilitate this evolution, I made key fields compulsory out at key stages. </div>
<div>
<ul style="text-align: left;">
<li>Acceptance Criteria had to filled out before something could be put in technical review </li>
<li>Story points had to be filled out before the story could leave Technical Review. </li>
</ul>
To do this in JIRA: </div>
<div>
<ul style="text-align: left;"><ol>
<li>Select to edit the workflow</li>
<li>Select the relevant transition and then the view validators option.</li>
<li>Select validators tab</li>
<li>Select to add a new validator</li>
<li>Select "required fields". Select the fields you want completed before the transition can happen</li>
</ol>
</ul>
<h3 style="text-align: left;">
Could the right stakeholder please stand up?</h3>
<div>
As stated, a goal of this process was to bring in the right stakeholder at the right time. More specifically:</div>
<div>
<ul style="text-align: left;">
<li>when the JIRA made it to UX Design it would be assigned to the team designer</li>
<li>when the JIRA made it to Technical Review - it was assigned to the technical lead. I played this role and then would either do the technical review or assign it to another senior technical person on the team. </li>
<li>If the issue didn't pass Technical Review and instead went back to Story Definition it was reassigned to the the issue reporter. </li>
</ul>
To enable the auto assign in JIRA, I added a "post function" to specific steps to auto assign the JIRA issue to correct stakeholder. This feature is available off any transition in the workflow. Just pick the transition where you want to auto assign. </div>
<h3 style="text-align: left;">
Stories, New Features or Improvements?</h3>
</div>
<div>
JIRA, out of the box, gives the option to create many types of issues amongst them: <i>Stories, New Features, Improvements.</i>
The problem is that there is an element of subjectivity here. Implementing something like
"Allow the user reset their password" could be a <i>Story </i>to one person, a <i>New Feature </i>to someone else and even
an <i>Improvement </i>to the person sitting next to them. I decided for this process, that something starts off as a "New Feature" or an "Idea" only. Then, it must go through the new requirements process and when done will be moved to either a Story or Epic. </div>
<div>
<br /></div>
<div>
This meant, that it made it easier to distinguish whether something was in Dev or still in a requirements capturing phase. When someone is searching issues in JIRA, when they see a Story or an Epic, they immediately know that it has gone
through the Requirements gathering process, it has been vetted and can then be confident it is really ready for development. This is really useful for backlog grooming which I think always works better when it is just a scheduling process rather than a debate about requirements that is open to the floor. A backlog grooming session that turns into a deep discussion about what should happen can just mean the entire dev team at a very long talk shop.</div>
<h3 style="text-align: left;">
Boards </h3>
<div>
As stated, we already had a Developer sprint board. This board worked well. We used it on the big screen at daily stand ups.
I wanted something similar for the requirements capturing process. I decided against changing the existing dev board, for the following reasons: </div>
<div>
<ul style="text-align: left;">
<li>It would would have way too many swimlanes (min 10) and difficult to even see on our massive widescreen. </li>
<li>The dev process and requirements process are really for two different sets of people. </li>
<li>The dev board was a scrum board. While our dev team was a mixture of Scrum and Kanban, we could use a scrum board, but there was no way the requirements team could. At the end of the sprint, everything was in the right most swimlane and the release was ready to go. It was nice to just have that board focussed on development with the end goal of releasing. Rather, than cluttering it with requirements that could evolve at a different pace. </li>
</ul>
<div style="text-align: left;">
So with that in mind, I created a new board specifically for the requirements gathering process. With the following swimlanes:</div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEihGsiOrCAuAJFNkXGfb3dEXCFh6UDxAOKGQClh39jw2FrI87xTxYyDkmV0u2U6x7w5F1NkwfpijFOMoz-t3bFCbqZ6blJOYkDYDo4R5KayrREBhM-H4e-htmV9kM9LrAIGoGJ3yplJ_R_W/s1600/Screen+Shot+2016-05-09+at+20.52.56.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="40" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEihGsiOrCAuAJFNkXGfb3dEXCFh6UDxAOKGQClh39jw2FrI87xTxYyDkmV0u2U6x7w5F1NkwfpijFOMoz-t3bFCbqZ6blJOYkDYDo4R5KayrREBhM-H4e-htmV9kM9LrAIGoGJ3yplJ_R_W/s640/Screen+Shot+2016-05-09+at+20.52.56.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Requirements swim lanes.</td></tr>
</tbody></table>
<div style="text-align: left;">
To avoid createing sprints for this board (they would be pointless), I made this a Kanban board. </div>
<h3 style="text-align: left;">
Filters</h3>
<div>
I added filters for key people on the Product side of the team. They were the individuals who would be driving most of the requirements. This was to help them track their own work and show what they were doing to the rest of the team.</div>
<h3 style="text-align: left;">
Priorities</h3>
<div>
I toyed with the idea of another field to give the JIRA issue a priority. However, why re-invent the wheel. JIRA already had a priority field which had 5 different levels (in order):</div>
<div>
<ul style="text-align: left;">
<li>Blocker</li>
<li>Critical</li>
<li>Major</li>
<li>Minor </li>
<li>Trivial</li>
</ul>
<div>
So, thought, let's just use this and if we need a 6th we can add. Five priority levels should really be enough though. To encourage people to use them, I colour coded the levels and made their colours display on the JIRA Board.<br />
<br />
This meant any time there was a discussion about all the stories, the priorities were obvious. Things could be reprioritised easily. The colour coding can be configured on the Agile board. I got some great feedback on what the various colours should be from the team! Naturally, I updated the existing Developer board to use the same scheme.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgfOZzsqRWkqInnvuyIQDkW1MS1Lurteb9k7wq-r4gIJu7EGF5AgWAFENX_UiWo7QDEfRkiQGD3I_ANk9h-MeiaPnfGdwO4ZFrIXM_J8rSZ9pRArBRngeM_tMijb86FB1u0S9TnkTQa9w1F/s1600/Screen+Shot+2016-05-09+at+20.58.51.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgfOZzsqRWkqInnvuyIQDkW1MS1Lurteb9k7wq-r4gIJu7EGF5AgWAFENX_UiWo7QDEfRkiQGD3I_ANk9h-MeiaPnfGdwO4ZFrIXM_J8rSZ9pRArBRngeM_tMijb86FB1u0S9TnkTQa9w1F/s640/Screen+Shot+2016-05-09+at+20.58.51.png" width="640" /></a></div>
</div>
</div>
<h3 style="clear: both; text-align: left;">
Meetings</h3>
<div>
Every process needs meetings. The requirements capturing process has three types of meetings:</div>
<div>
<ul style="text-align: left;">
<li><b>Key Technical Leads and Product Analysts met once a week</b>. We would run through the entire requirements board, spending two mins on each issue. I used this opportunity to make sure nothing was blocked, we were in agreement on priority and everyone was buying into the process. It is important to emphasis this is not backlog grooming session. All requirements must go through this process first before they are ever discussed in backlog grooming. </li>
<li><b>Product Analysts met amongst themselves</b>. This meeting didn't involve me but it gave the Product side a chance to delve more into stories, exchange ideas and do important work without having to take away time from Technical people. </li>
<li><b>Product analyst and Developer for Story</b>. Usually, each issue ended up with one Product analyst and one Technical stakeholder who would end up developing the feature. They could met 1-1 whenever they saw fit. This was to thrash out requirements, tease out edge cases, you name it. The idea here was to give the developer a sense of ownership of the feature. They were involved from an early stage and would responsible for delivery of the new feature. How they met, how often they met was up to them. </li>
</ul>
<h3 style="text-align: left;">
My Role</h3>
</div>
<div>
For this project I was a mixture between a Tech Architect, a Tech Lead and also coded. To do the Tech Architect and Tech Lead work, I generally avoided stories on the critical path and didn't code as much as some of the other developers on the team. I initially was going to do all the Technical Reviews. I decided against this as it just doesn't scale and it would slow things down. I also wanted to get developers involved early and give them a sense of ownership. The team had strong developers who wanted full stack development. The best way to achieve that is to give them a feature to own and then just provide the architectural advice when needed. The architectural advice would be on the lines of:<br />
<br />
<ul style="text-align: left;">
<li>Discussion and agreement of changes to key interfaces, ReST endpoints or Database schema</li>
<li>Patterns</li>
<li>Best practises</li>
<li>Technical risk assessment and mitigation</li>
</ul>
<br />
When the issues made it to Technical Review, they were auto assigned to me in JIRA and then I would delegate them out as much as possible and then just zoom in and out as appropriate.<br />
<br />
<h3 style="text-align: left;">
Ah more advantages</h3>
</div>
<div>
The other advantage of this process was that I could see what might be coming up in the next 6 months. Product Analysts would continuously add ideas for things that might or might not evolve into an actual requirement depending on customer needs etc. This meant, that I could be confident that the existing architecture was safe with respect to what might be coming for next 6 months. So for example, if I saw a lot of ideas coming in around different types of Searching, it would mean I would keep something like Elastic Search on the Tech Radar - as in this might become more relevant very quickly at some stage. It would also help strike a good balance in technical decisions and avoid over engineering for scenarios that were extremely unlikely. </div>
<h3 style="clear: both; text-align: left;">
Final words</h3>
<div class="separator" style="clear: both; text-align: left;">
Anyway, that's about it. I believe no matter what you are doing, to get the best results you need a good structure. In software, that means you need a good process. Everyone needs to know what they need to be doing, what the priority is and the structure should make it as easy as possible for the team (which is always going to be a mixture of skills and backgrounds) to work together to deliver a top quality product. </div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
</div>
<div>
<ul style="text-align: left;">
</ul>
<div>
<br /></div>
</div>
<div>
</div>
</div>
Alex Staveleyhttp://www.blogger.com/profile/04378245339254378938noreply@blogger.com2tag:blogger.com,1999:blog-8965734277283465944.post-60112820693558968592016-02-07T12:39:00.003-08:002020-04-25T10:14:39.271-07:00Book Review: Building Microservices <div dir="ltr" style="text-align: left;" trbidi="on">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgEumJdVDfyoas9FUcXct4LfSRsQ6kHaCKCnVdVYLf1H5kmz8x5Bh52OA1c4B28gJqS-akV648OcEB_VUNqqgjG7NJuJonjB9uD0rqfs-oPlQ5Mz4McJBwOBNNRmtILsSPFQoG9kPxmUh4T/s1600/lrg.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgEumJdVDfyoas9FUcXct4LfSRsQ6kHaCKCnVdVYLf1H5kmz8x5Bh52OA1c4B28gJqS-akV648OcEB_VUNqqgjG7NJuJonjB9uD0rqfs-oPlQ5Mz4McJBwOBNNRmtILsSPFQoG9kPxmUh4T/s200/lrg.jpg" /></a>The architectural idea of microservices was inspired - in part - by Unix's philosophy of code being short, simple, clear, modular, extendable and
that it could be repurposed by developers. The term is currently up there with Internet of Things, Big Data and the Cloud,
in contemporary technical lexicon.
<br />
<br />
Author <a href="http://samnewman.io/">Sam Newman </a>is an industry expert on the subject. He has:<br />
written for <a href="http://www.infoq.com/presentations/Micro-Services">Infoq</a>, presented at <a href="https://vimeo.com/105751281">Javazone</a> and various <a href="http://samnewman.io/talks/principles-of-microservices/">other events</a> regarding microservices. <br />
<br />
His book '<i><a href="http://www.amazon.co.uk/Building-Microservices-Sam-Newman/dp/1491950358/ref=sr_1_1?ie=UTF8&qid=1454877952&sr=8-1&keywords=building+microservices">Building Microservices'</a> </i>does an excellent job introducing the key concepts of microservices:<br />
<ul>
<li> services are autonomous, live on their machines
</li>
<li> resilience - one service fails, it doesn't impact other services
</li>
<li> scaling - because services are independent they can be independently scaled
</li>
<li> deployment - deployment should be easy. A change to a service means that's all that's deployed.
</li>
</ul>
It is worth pointing out it isn't just a book about microservices. Many of the ideas and best practises detailed (for example HATOES in your ReST approach and to check OWASP for security references) are equally applicable to non - microservices architectures. But, that said, what I really like about this book is anytime a general architectural or software engineer concept (and there are a lot) is explained it is explained very well. Some examples:
<br />
<ul>
<li> Reactive extensions (running operations as a result of multiple calls not caring if they were blocking / non blocking. For ref: see <a href="https://msdn.microsoft.com/en-us/data/gg577609.aspx">here</a> and <a href="http://reactivex.io/">here</a>). </li>
<li><a href="http://martinfowler.com/bliki/CatastrophicFailover.html"> Catastrophic Failover</a> </li>
<li><a href="https://en.wikipedia.org/wiki/Robustness_principle"> Postel's law</a> </li>
<li><a href="http://martinfowler.com/bliki/TolerantReader.html"> Tolerant Readers</a> </li>
</ul>
There are - of course - many tips regarding how you approach microservices. For example:
<br />
<ul>
<li> Don't get too hung up on DRY. It may make it harder to keep your services independent.
</li>
<li> Consider using CDC's for your testing approach
</li>
<li> Canary releasing / Blue, Green testing for releasing.
</li>
<li> Using bulkhead and circuit breaker patterns to make services ore resilient.
</li>
</ul>
As an overview to microservices, it's a great book. However, the reader must bear in mind there is no one size fits all approach to microservices.
The finer details will depend on your project, your team and even a bit of trial and error. It's difficult to critique this book. So instead,
I'll just flag some concerns <i>not about the actual book</i> but about microservices in general and how the industry and developers are reacting to them.<br />
<br />
My first concern with microservices is more a practical than a technical one. Doing modular design for everything from your
schema, service layer, end points, configuration isn't as easy as some people might think. Especially in teams of varied skill level,
varied backgrounds and inevitable commercial pressure that happens in every project. It requires a lot of technical aptitude, leadership and discipline.
If a project is not good at achieving modular design in a monolith - for whatever reason - I think it will really struggle at modular design in microservices when the complexity of the network has to be also considered.
<br />
<br />
The second concern I'd have is that when explaining the principle ideas of microservices, it is often compared with a monolith to the point the word monolith is a pejorative term. The two approaches: monolith and microservices are presented as if it's either one or the other. I think this a <a href="https://en.wikipedia.org/wiki/False_dilemma">false dichotomy fallacy</a>. There are other approaches available. <br />
<br />
Thirdly, microservices are no doubt, a clever idea. But that doesn't mean there are a panacea. In some projects they will be a good fit, in others they will not be worth it. One obvious factor to bear in mind is your non-functional requirements. One useful point of reference, that is worth considering is the project <a href="https://www.thoughtworks.com/profiles/james-lewis">James Lewis</a> (another <a href="https://www.thoughtworks.com/">Thoughtworks</a> guru) described in his talk about <a href="http://2012.33degree.org/pdf/JamesLewisMicroServices.pdf">microservices in 2012</a>. In this presentation, three non-functional requirements for the project caught my attention:<br />
<ul>
<li>One component had to handle 1,000 TPS
</li>
<li>Another had to support a user base of 100 million users
</li>
<li>Another had to support batch loads of 30 - 90 million records
</li>
</ul>
While I am not saying you should be in this ballpark before considering microservices, I am just trying to suggest that most projects don't have these sort of demands and there's a lot of merit of considering something like a very well structured modular monolith first using DDD principles and then to migrate towards microservices should the benefits justify it. This strategy is well explained by Martin Fowler in his <a href="http://martinfowler.com/bliki/MonolithFirst.html">Monolith first</a> article.
<br />
<br />
Some other references:
<br />
<ul style="text-align: left;">
<li>Microservices <a href="http://www.infoq.com/articles/seven-uservices-antipatterns">Anti-Patterns</a></li>
<li>Martin Fowler, <a href="http://martinfowler.com/articles/microservices.html">Microservices</a></li>
<li>James Lewis microservices talk at <a href="http://www.infoq.com/presentations/Micro-Services">Infoq</a> </li>
<li>James Lewis microservices <a href="http://www.se-radio.net/2014/10/episode-213-james-lewis-on-microservices/">Software Engineering Radio podcast</a></li>
<li>James Lewis 33 Degree conference http://2012.33degree.org/talk/show/67</li>
</ul>
<br />
<br />
Until the next time enjoy yourselves.<br />
<br />
<br /></div>
Alex Staveleyhttp://www.blogger.com/profile/04378245339254378938noreply@blogger.com1tag:blogger.com,1999:blog-8965734277283465944.post-78487435437699575102016-02-07T12:13:00.003-08:002016-02-07T12:13:57.291-08:00Fail Safe / Fail Fast <div dir="ltr" style="text-align: left;" trbidi="on">
When developing a rapid prototype it can make sense to put the emphasis on the '<i>happy path</i>' and not consider things like exception handling, edge cases and failure. Perhaps, once the prototype phase of the project is over the code will be thrown out or it will be refactored to deal with the real world.<br />
<br />
In a production system we simply don't have luxury to only consider the happy path. The more production system I have worked on the more types of failure I have being exposed to - some of it <i>very painful</i>. As my hair went greyer from these experiences, I couldn't help thinking more and more about how to make failure less painful. Who wants their pager going off for some silly reason? Nobody. If dealing with production code, we simply must think about how best to deal with various forms of failure. <br />
<br />
In this <a href="https://prezi.com/_wo62nqopecq/software-engineering-failure-techniques/">presentation</a> I consider two engineering techniques that I think should always be on the architectural radar:<br />
* Fail fast<br />
* Fail safe<br />
<br />
<br /></div>
Alex Staveleyhttp://www.blogger.com/profile/04378245339254378938noreply@blogger.com1tag:blogger.com,1999:blog-8965734277283465944.post-16677045961745400592016-01-24T14:28:00.001-08:002016-01-25T11:38:55.966-08:00Developer Productivity<div dir="ltr" style="text-align: left;" trbidi="on">
Recently I gave a <a href="https://prezi.com/nebe2gr8efox/developer-productivity/">presentation</a> regarding developer productivity - something that has always interested me. Generally, when a system gets out of control with spiralling technical debt and software entropy it becomes impossible to make changes or add new functionality to that system in a reasonable and predictive way. At a critical level, this can be the difference between project success and project failure. If processes are smart, good structures are in place, developers are productive not burning out from late hours debugging crazy code, terrified of the impacts of any change. <br />
<br />
Everyone wants a path to project delivery that is smoother. No matter what the job or the task it is generally far easier to get a sense of satisfaction from a sense of productivity. But let's face it; software entropy exists in nearly every project. Why? Are we not smart enough to use the machines we designed? Do commercial realities inevitably mean bad code happens more than good code? <br />
<br />
These are questions that spark lively discussions. In this presentation, I outline some of my own ideas on <a href="https://prezi.com/nebe2gr8efox/developer-productivity/">developer productivity</a>.<br />
<br />
<br />
<br /></div>
Alex Staveleyhttp://www.blogger.com/profile/04378245339254378938noreply@blogger.com0