What's the case for your API?

Disclaimer: In pure REST, API's are opaque and the URL should be whatever what was sent as a link in the response to a previous request. But, I'm not talking pure REST, I'm talking more pragmatic APIs which involve some concepts from REST and as well as general API best practices. 

When writing an API, it starts simple. You identify the obvious resources and end up with endpoints such as:

/api.mycompany.com/tweet

Eventually, your API will have to capture more sophisticated concepts and model more complex resources that cannot be expressed in short single nouns.  Some real world examples include:

• Enabling request validation via a Request Validator resource (AWS API Gateway API)
• Performing a customer search via a Customer Search resource (Google Customer Search API)
• Running powerful checks against code via a Check Runs resource (Github API)

In English grammar, nouns that are really two nouns joined in some way are called compound nouns and in English grammar, compound nouns follow one of three patterns:

1. All the one word: haircut, toothpaste
2. Two words: rain forest, ice cream
3. Hyphenated: self-esteem, brother-in-law

In the API world there are different options to choose from but it is better for consistency that your APIs just pick one approach and stick to that it. So firstly, what are the options for compound nouns from an API perspective?

Camel Case

Camel case is the practise of writing each word in the phrase with a capital letter.  There are two variations:

1. Initial upper case (also know as Pascal's case) is where the first letter is also a capital, for example: IceCream.  Pascal's case is popular in programming languages for naming classes e.g. Java. 
2. Initial lower case is where the initial letter is always lower case, for example: iceCream.  This approach is popular in programming languages (again Java is a good example)  for naming variables.  When people say camel case, they are usually referring to the initial lower case format.

Kebab Case

In Kebab Case, the individual words are separated by hyphens. Ice cream is expressed as ice-cream.  This approach is used in the Lisp programming language, in lots of URLs (for example, every blog post in www.blogger.com e.g. http://dublintech.blogspot.com/2018/08/oauth-20-authorisation-code-grant.html).  The observant amongst you will note sometimes the word "dash" is sometimes used in technical references instead of "hyphen".  So, what's the difference?  In English grammar, hyphen is the thing  used to join two words to make one whereas the dash is the thing used to usually add some sort stylistic emphasis to the end of a sentence such as: "I might have an interesting point here - you never know".

In programming we don't care whether the term is hyphen and dash. They are used interchangeably and mean the same thing.

The kebab case approach became popular in Web URIs because search engines knew that the hyphen meant separate words and could index the URI properly.  This convention used by search engines meant hyphens became a de facto standard for URIs.

Snake Case

In this approach, an underscore is used to separate words.  Ice cream becomes ice_cream. This approach is used in Python and Ruby for anything other than a class name or static constant.

Join words

In this approach the words are just joined. There is no -, no _ and no capitalisation of anything. This is not a popular approach with developers because it's difficult to read.

APIs

Should we use camelCase, kebab-case or snake_case in an API?  Well unfortunately, Mr. Fielding's dissertation did not go into such detail.  So what are people actually doing?  And is the approach used consistent across the API's URL and the JSON Body.  Let's take a look.

AWS

AWS have different API styles for different services.  The API Gateway REST API reference shows that JSON payload uses camel case but the URL uses nothing, it's just:

/restapis/{id}/requestvalidators/{requestvalidatorId}

Google

Surprise, surprise Google also have lots of APIs. The Google

Custom Search API is similar to the AWS API Gateway API.  The compound noun in the URL is just the one word and the JSON body is camel case.  

The Google Gmail API has camel case in request body and in some URLs, for example the forwarding addresses API.  

The Google youtube API sometimes will use kebab case in the URL e.g. yt-analytics but in other cases will use single word e.g. youtubepartner.   But the JSON payload is camel case.

Github

The Github API is a good example where we get a reminder that if possible, you should try to avoid this issue by trying to avoid compound nouns as it avoids them by using some creative name spacing.

However, some more rooting around and you'll find a compound noun such as check run expressed using kebab case in the URL and he JSON body using snake case. 

Stripe

Stripe use snake case in the URL and in the JSON body.  For example the PaymentsIntents API. 

 https://api.stripe.com/v1/payment_intents 

and JSON body...

{
  "id": "pi_Aabcxyz01aDfoo",
  "object": "payment_intent",
  "allowed_source_types": [
    "card"
  ],
  "amount": 1099,
  "amount_capturable": 1000,

Paypal

Paypal have more compound nouns than the other APIs checked. APIs for resources such as billing agreement  the API will use kebab case in the URL but then use snake case in the JSON payloads.

Twitter

Twitter use snake case in the URL e.g. /saved_searches/ and snake case in the JSON payloads.

Facebook

Facebook's Graph API tends to avoid resource naming in URLs and in JSON bodies it is snake case.

By this stage, you should be getting a little but confused. So let's recap via the table below.

API	URL	JSON Body
AWS API Gateway	No separator	camelCase
Facebook Graph API	N/A	snake_case
Github	Snake and Kebab	snake_case
Google custom search	No separator	camelCase
Google Gmail	camelCase	camelCase
LinkedIn	camelCase	camelCase
Pay pal	kebab-case	snake_case
Stripe	snake_case	snake_case
Twitter	snake_case	snake_case

Everyone is different, what should I do?

So there is a lack of consistency across the industry.  However there are point worth making:

1. In general compound nouns are best avoided.  In all the APIs checked (except PayPal), they appear in under 5% of the APIs.  This means developers don't get upset when their favourite approach is not used.
2. The only Web API in the selection above that had more than 5% of its APIs using compound nouns was PayPal and they went for kebab-case in URIs.
3. kebab-case is never used in any JSON body.  The syntax is allowed.  So what drives this trend? It's more than likely because JavaScript Web UIs are possibly the mos popular client invoking API and the similarly the most popular back end language serving the API is Java and both of those dudes don't allow the - in any of their declarations.

Is there anyone else saying anything?

In the excellent REST API Design Cookbook, industry expert Mark Masse suggests:

1. Make your APIs lower case, which rules out camel case
2. Use kebab case when expressing compound terms in your URIs
3. Avoid using underscores as they may not display well since some browser render hyperlinks with an underline added

Make a decision

1. Avoid compound nouns if you can.  This isn't always possible. Sticking to a ubiquitous language is important and helpful.  If you have a complex business application you will have lots of compound nouns. 
2. If you can't avoid compound nounds and more than 5% of the APIs are going to involve compound nouns use kebab case for your URIs. Why?  Because if you have a complex business domain it's not only developers you need to think about.  Lots of BA's, Product Architects, curious Managers will also be looking at your APIs. Kebab-case is the easiest to read for everyone.
3. For JSON body, I think it is okay to use camelCase because this is the easiest to map back to JavaScript and Java code.  It is also a recommendation from Google to use camelCase in JSON.
4. If you have to use camelCase in your URI's, consider using the first letter capital approach for the URIs as the URIs are supposed to marking resources not attributes. Resources are more analogous to Java Classes which also use initial letter capital format; whereas the JSON payload attributes are analogous to Java attributes which use initial lower case.

Until the next time, take care of yourselves.

Technical Debt will kill your Agile Dreams

Warning this blog post is a bit of a rant, if you want something more technical there's plenty of other articles on this blog and many others!

Technical debt is a concept used in software engineering to express the additional complexity that is added to a project due to technical decisions that result in inferior solutions being chosen because they can be delivered quicker. It is analogous to financial debt. Borrow money for some gain, incur debt but -and there's always abut - you have to pay the debt off. Otherwise, with interest, the debt will grow and you will eventually be bankrupt.  Similarly, it is ok to incur some technical debt.  If you spend too long looking for the perfect solution your customers will have moved over to someone else. At the same time, if you incur too much technical debt and you are not meeting your repayments, just as the interest on the financial debt will compound repayments, if you don't meet your technical debt repyaments the complexity and entropy increases with time and eventually it will stagnate your product. Something that should really only take one month to deliver, suddenly takes 3 months, then 6 months, then there's only a few people in the entire company who can do it in 6 months, then a few of them leave, then...

Where the analogy breaks down is that all too often technical debt is also used to express that there has just been some bad engineering. Bad engineering decisions are in a different category to ones that were tactically made with full knowledge that the short-term priority was worth it. When it's clear that such a decision was, in fact, a tactical decision, it is much easier to convince people that refactoring needs to happen and the debt has to be paid off. Unfortunately, when the term is used as a polite way of saying bad engineering, it's unlikely there is any repayment strategy in place and it is even harder to create one because first, you need to convince people there is some bad engineering, then you need to convince people it is causing problems, then you have to have the ability to think of a better approach, cost it and then convince the various stakeholders the investment is worth it.  It is like trying to win 5 matches in a row away from home when the odds are against you.  

So, what are the signs that your technical debt (irrespective of whether it is intentional or accidental) is just too high and you could be filing for bankruptcy soon?

Well, think about this. In the Agile world, we want to develop, build, release in quick cycles and get rapid feedback from the customer and go again. This is only possible if there is an abundance of high quality and well engineered automated tests that run fast and provide confidence that a change no matter what it is has not broken anything. It doesn't matter if the breakdown of the tests are: 68% unit tests and 32% are integration tests or 91% are unit tests and 9% integration tests, the tests have to run fast and they have to provide confidence to all stakeholders. Otherwise releasing will be a pain and it will not be possible to release regularly. That means being Agile and getting all the benefits of it will be very difficult — no matter how good your backlog grooming sessions are.

What usually makes it difficult for developers to write good tests? Well, it's usually technical debt. It doesn't matter if it is intentional or accidental.

Now, there are all sorts of tools that will measure technical debt and put it up on a nice looking SonarQube board but usually these tools only pick up the trivial stuff — removing an unused import etc. Who cares?  Such trivial stuff isn't going to slow anyone down.   The real technical debt problems are the ones that slow people down, they make it harder to make changes, fix bugs, add functionality and do it all quickly with confidence - why because they have made things much harder to test.  Sadly, these are technical problems are usually not just something an IDE, a PMD or Checkstyle will prompt you to do.  They are generally much deeper in nature - towards the architectural end of specturm. Some examples:

• Lack of data encapsulation and immutability leading to huge cyclomatic complexity, unpredictable code paths and difficulty in predicting impacts.
• Impedence mismatches
• Lack of modularity and too much coupling
• Lack of or bad application of patterns
• A proprietary language introduced with no IDE support, no mechanism to easily unit test or debug code. Zero support from Stackoverflow.
• Spawning of threads and asynchronous call paths when are better approaches which would be much easier to test

This is where the analogy of technical debt and financial debt breaks down.   When you have architectural debt you have big problems. When you have a HashMap used when an ArrayList would have made more sense, you don't. Financial debt doesn't have such a critical distinction.  Unless we say, the debt is due to a friendly and sympathetic bank you have a good relationship with or its debt due to some lunatic load shark who will call around to your house with a baseball bat.  

So, if you realize you are approaching your credit limit, what do you do? Firstly, you need to get confidence in your tests. They need to test key functionality and they need to be maintainable. That is more important than speed. If you can't get confidence, you can't ship. There is not much use with tests that run in 5 minutes if no-one has any confidence the functionality you deliver will actually work. Secondly, once you have confidence in the tests (even if they are ugly end-to-end tests), you need to get them to run fast. This is where you can start refactoring — the end to end tests should facilitate refactoring of parts of the call paths; there may be obvious sprouts for example and this should help to move towards a classical test pyramid. 

Thirdly, you now need to understand why the code is so difficult to achieve high-quality tests. Too much coupling, bad exception handling, bad decompositions it's probably a long list. Understanding why your code is difficult to test is a key architectural and engineering skill as it requires the ability to not just understand the complexity but the ability to be able to know how to reduce it.   Reducing the debt should then provide a pathway to make the code easier to test and thus make good tests run fast.   Achieving this means you are winning the battle against eventual project stagnation. 

So, the last part of the rant.  There is a growing problem with the application of Agile which means we end very easily end up with faux Agile. The various Agile books, courses, blogs will detail things like story points, burn downs, stand ups. That stuff is all good but there is not enough focus on technical excellence. Without the technical excellence you get inevitable architectural problems and code that is difficult to test irrespective of your best efforts at story pointing, backlog grooming, sticking yellow post-its up on walls, having arguments over the definition of done, doing your show and tells. All that stuff is nice and beneficial but in comparison to technical excellence it is almost superficial. It is something people can easily see but on its own it never captures the complexity of technical debt.  It is the ratio of technical debt to technical excellence that determines whether you can write testable code easily and thus be able to deliver in regular short iteration which is the goal of Agile. Isn't it? 

Lastly, something we all say to our kids when they try any sport: "If you are losing kido, never give up, play to the end of the match". 

Technical Debt is natural, it happens in every project in the world.  Your job is to keep it in check and when it gets too high to be able innovate your way work around it and eventually lesson it.  That's such an essential characteristic of a good Architect, it is also a handy interview question. 

"Mr. Candidate, describe some Technical Debt you have experienced, describe the effects of it and what strategies you put in place to either work around it or deal with it?"

Or even, 

"Mr. Candidate here are some examples of Technical Debt, how would you prioritise which ones to deal with? Could you describe some better solutions and how would you convince people they are worth it?"

The quality of detail in the answer you get back is very likely to indicate the level of pragmatism, experience and the innovation skills of the Architect /  Engineer.

Until the next take care of yourselves.

OAUTH 2.0 Authorisation Code Grant

OAuth 2.0 provides a number of security flows (or grant types) to allow an application access user's data in another application.  In this blog, we will look at the
OAuth 2.0 grant: Authorisation Code Grant.

Firstly, a number of definitions:

• Client: The application the user is currently interacting with.  For example, let's assume a fictitious funky blogging site: www.myfunkyblog.com.  The client wants to communicate with another application and retrieve something about the user from there.  For example, their favourite photo!  Let's assume the fictitious megaphotosharing.com as the service the client wishes to access. 
• Client ID: this is an ID which identifies the client.  It can be passed around openly in Web URLs etc
• Client secret ID:  A secret ID that only the client knows. This is kept server side and will be used in requests to the application for which access is sought.   It cannot be passed around in Web URLs. 
• Resource owner: This is usually the human, who is using the client application. The resource owner has data in another application (e.g. megaphotosharing.com) that the client (myfunkyblog.com) wishes to access. The goal is to facilitate that sharing without the need for the Resource owner aka the human to ever pass their megaphotosharing.com password to myfunkyblog.com.  Note: the resource owner doesn't have to be a human but interestingly according to the OAuth spec, when it is a human it can also be referred to as the end - user. 
• Resource Server: hosts the protected resources of the resource owner that the client is interested in. So this is the megaphotosharing.com server that has the Resource owner photos the myfunkyblog.com is interested in.
• Authorisation Server: the server who issues a token to myfunkyblog.com after the resource owner has successfully authenticated and allowed myfunkyblog.com obtain some of its megaphotosharing.com.  Sometimes the Authorisation Server and the Resource server are effectively the same but they don't have to be. 
• Access token: a special type of token the myfunkyblog.com authorisation server gives megaphotosharing.com to access the protected resources. It will contain scope, lifetime and other access attributes.

Use case

So the use case is that the client (myfunkyblog.com) wants to access information about
the resource owner (the human) from another application: megaphotosharing.com.

Client Registration

The first the client must do is register with the service (megaphotosharing.com)
providing its name, website etc.  The service will return a secret client code.
The client keeps this private and is responsible for ensuring only it knows it.  Usually,
it will encrypt and persist it in the client somewhere on the backend.  The service will
also receive a client id.  Unlike the client secret, this is public and can be passed around
in URLs etc.

Flow

Ok so now the actual flow.    The user is browsing around myfunkyblog.com and accesses a part of the site where myfunkyblog.com wants to know what the end-user's favourite photo is.

A pop-up screen appears to end-user.

This has URL:

https://megaphotosharing.com/oauth/authorize?response_type=code&client_id=CLIENT_ID&redirect_uri=CALLBACK_URL&scope=read

Key parts of this URL:

• megaphotosharing.com: This is the domain for the authorisation server
  
• response_type=code: Required parameter to enable the client informs the authorization server the desired grant type. An alternative value would be the "token", this is for the implicit flow.
  "code" means the client wants an authorization code which will be returned after
  resource owner logs in.  This authorization code will be used in a subsequent request by the Client. 
• client_id: Required parameter, to identify the client.  Remember this is public and
  can be passed to and from a web browser.
  
• redirect_uri: This is an optional parameter.  It enables the client to dynamically specify the URL the auth server should redirect to.  In some flows, this isn't needed as there is only one redirect URI and this is registered by the client with the service during client registration.
  
• scope:  This is an optional parameter.  It specifies the level of access that the application is requesting. In this case it is just a read.  The auth server uses this to inform the user / resource owner what the client is trying to do.

The user then logs into megaphotosharing.com who tells the user what the Client wants to do.  If the user selects ok, megaphotosharing.com redirects to the redirect URI passed up.

https://myfunkyblog.com/callback?code=212132kjhkhj

Notice how the client id is passed over the web in a URL and the authorisation code is passed back over the web.

The client, then uses the returned authorization code, its client id, client secret and the grant type to make a POST request Server to Server to get an Access Token.  This happens all on the back end.

https://megaphotosharing.com/v1/oauth/token?client_id=CLIENT_ID&client_secret=CLIENT_SECRET&grant_type=authorization_code&code= 212132kjhkhj&redirect_uri=CALLBACK_URL

Notes:

• client id and client secret identify the client.  This is a back-end request and hence it is okay to pass the client_secret (which would obviously never pass to or from the browser).
• grant_type: This must be set to authorisation_code. As it indicates the Authorisation Code Grant.  Remember, the grant is used to indicate the flow the client is using (it can also be used by the server what types of flows are available). If the client was using the Client Credentials Grant, this value would be: "client_credentials".  If the client was using "Resource Owner Password Credentials Grant" the value would be "password".
• code: 212132kjhkhj - The actual authorisation code what was returned from initial authorisation request from the authorisation server. This is required.
• redirect_uri: if the redirect_uri was included in the authorisation request this value must be the same as the value used in that request.

The client then receives back an access token. Something like this:

{"access_token":"ACCESS_TOKEN","token_type":"bearer","expires_in":2592000,"refresh_token":"REFRESH_TOKEN","scope":"read","uid":1001013121222}

It will now use this to access some of the resource owner's resource data.

So what's the big deal?

• There are obviously big advantages for users not having to tell one website its password for another site. 
• Reduces the number of passwords a user needs to remember
• Allows richer websites by allowing disparate applications to talk to each other. 

Why do people find it confusing?

There are a number of reasons why people find OAuth 2.0 confusing. 

• There are a few different flows or grants.  The Authorisation Code Grant is just one.  Sometimes when you google explanations for OAuth 2.0 you get explanations for different grants without making it clear what is and isn't being explained. Hence why I put Authorisation Code Grant in the title.  
• Terminology.  I'll just speak for myself.  But if I am reading quickly, I am likely to:
•  confuse "Client" with the end-user
• get confused between the Resource Server and Authorisation Server
• Consistenty.  A lot of places implement OAuth 2.0 or something very similar to OAuth but will refer to things differently along the way.  For example, go to quora.com and try to login to google.  You are taken to: 

  https://accounts.google.com/signin/oauth/oauthchooseaccount?client_id=917071888555.apps.googleusercontent.com&as=rdWeinbqWJbt6ChoW2f3Fg&destination=https%3A%2F%2Fwww.quora.com&approval_state=!ChRyQlhnbEYzai1xQTliNlNmTEVmNRIfZ3doM2hlRVIycGdiMEVBN1JaNXdOM085MERXLVVCWQ%E2%88%99ANKMe1QAAAAAW2i2to0SOyO2_w3k3O4gjwUKQLGNmZ2h&oauthgdpr=1&xsrfsig=AHgIfE8EzSxvWfzyxou0dwLDxv4GhD6e5g&flowName=GeneralOAuthFlow
  

  There's no response_type in that URL.
• OAuth is an authorisation spec.  It is usually used with Authentication spec like Open Connect but that is actually a separate spec. 

Java Lambda Streams and Groovy Clouses Comparisons

This Blog post will look at some proverbial operations on List data structure and make some comparison between Java 8/9 and Groovy syntax.  So firstly, the data structure.  It's just a simple Rugby player who has name and a rating.

Java

class RugbyPlayer {
    private String name;
    private Integer rating;
    
    RugbyPlayer(String name, Integer rating) {
        this.name = name;
        this.rating = rating;
    }

    public String toString() {
        return name + "," + rating;
    }
        
    public String getName() {
        return name;
    }
        
    public Integer getRating() {
        return rating;
    }
}

//...
//...
List<RugbyPlayer> players = Arrays.asList(
    new RugbyPlayer("Tadgh Furlong", 9),
    new RugbyPlayer("Bundee AKi", 7),
    new RugbyPlayer("Rory Best", 8),
    new RugbyPlayer("Jacob StockDale", 8)
);

Groovy

@ToString
class RugbyPlayer {
    String name
    Integer rating
}
//...
//...
List<RugbyPlayer> players = [
    new RugbyPlayer(name: "Tadgh Furlong", rating: 9),
    new RugbyPlayer(name: "Bundee AKi", rating: 7),
    new RugbyPlayer(name: "Rory Best", rating: 8),
    new RugbyPlayer(name: "Jacob StockDale", rating: 8)
]

Find a specific record

Java

// Find Tadgh Furlong
Optional<RugbyPlayer> result = players.stream()
    .filter(player -> player.getName().indexOf("Tadgh")  >= 0)
    .findFirst();      
String outputMessage = result.isPresent() ? result.get().toString() : "not found";

System.out.println(outputMessage);

Groovy

println players.find{it.name.indexOf("Tadgh") >= 0}

Comments

• The Java lambda has just one parameter: player.  This doesn't need to be typed as its type can be inferred.  Note: If there were two parameters in the parameter list, parenthesis would be needed around the parameter list.
• In Java, a stream must be created from the List first before the functional operation can be applied.  
• A lambda is then used to before performing a function which returns an Optional
• The lambda definition doesn't need a return statement.  It also doesn't need {} braces or one of those semi-colons to complete a Java statement.  However, you can use {} if you want to and if you want to, you must include the ; and the return statement.  Note: if you lambda is more than one line, you don't get a choice, you must use {}.   It is recommended best practise to keep Lambda's short and to just one line. 
• Java 8 supports fluent APIs for pipeline Stream operations.  This is also supported in Groovy Collection operations.
• In Java a player variable is specified for the Lambda.  The Groovy closure doesn't need to specify a variable.  It can just use "it" which is the implicit reference to the parameter (similar to _ in Scala).  
• The Java filter API takes a parameters of type Predicate.   A Functional Interface means: can be used as the assignment target for a lambda expression or method reference.  Predicate, is type of Functional interface.  It's one abstract method is: boolean test(T t).    In this case, in the lamda, the player corresponds to t.  The body definition should evaluate to a true or a false, in our case player.getName().indexOf("Tadgh") will always evaluate to true or false.  True corresponds to a match. 
• Java 8 has other types of Functional Interfaces:
• Function – it takes one argument and returns a result
• Consumer – it takes one argument and returns no result (represents a side effect)
• Supplier – it takes not argument and returns a result
• Predicate – it takes one argument and returns a boolean
• BiFunction – it takes two arguments and returns a result
• BinaryOperator – it is similar to a BiFunction, taking two arguments and returning a result. The two arguments and the result are all of the same types
• UnaryOperator – it is similar to a Function, taking a single argument and returning a result of the same type
• Java 8 can infer the type for the lambda input parameters. Note if you have to specify the parameter type,  the declaration must be in brackets which adds further verbosity.
• Groovy can println directly.  No System.out needed, and no need for subsequent braces.
• Like Java, Groovy doesn't need the return statement.  However, this isn't just for closures, in Groovy it extends to every method.    Whatever is evaluated as the last line is automatically returned. 
• Groovy has no concept of a Functional interface.  This can mean if you forget to ensure your last expression is an appropriate boolean expression, you get unexpected results and bugs at runtime.
• The arrow operator is used in both Groovy and Java to mean effectively the same thing - separating parameter list from body definition. In Groovy it is only needed it you need to declare the parameters (the default it, doesn't suffice). Note: In Scala, => is used.

 

Find specific records

 

Java

// Find all players with a rating over 8
List<RugbyPlayer> ratedPlayers = players.stream()
    .filter(player -> player.getRating() >= 8)
    .collect(Collectors.toList());
ratedPlayers.forEach(System.out::println);

Groovy

println players.findAll{it.rating >= 8}

Comments

• In the Java version, the iterable object ratedPlayers has its forEach method invoked.   This method takes a FunctionalInterface of type Consumer (see Jdoc here).  Consumer, methods a function which takes an input parameter but returns nothing, it is void.  
• In Java, the stream.filter() will return another stream. Stream.collect() is one of Java 8's Stream terminal methods. It performs mutable fold operations on the data elements held inside the stream instance return by the filter method.  
• Collectors.toList () returns a Collector which collects all Stream elements into a List.
• When using the toList() collector, you can't assume the type of List that will be used.  If you want more control you need to use the toCollection().  For example: 
• .collect(toCollection(LinkedList::new)
• Note: We could have omitted the .collect() operation and invoked forEach straight on the stream.   This would make the Java code shorter.  

players.stream()
   .filter(player -> player.getRating() >= 8)
   .forEach(System.out::println);

• System.out::println is a method reference - a new feature in Java 8.   It is syntactic sugar to reduce the verbosity of some lambdas.  This is essentially saying, for every element in ratedPlayers, execute, System.out.println, passing in the the current element as a parameter.
• Again less syntax from Groovy.  The function can operate on the collection, there is no need to create a Stream.  
• We could have just printed the entire list in the Java sample, but heck I wanted to demo forEach and method reference.

 

Map from object type to another

Java

// Map the Rugby players to just names. 
// Note, the way we convert the list to a stream and then back again to a to a list using the collect API. 
System.out.println("Names only...");
List<String> playerNames = players
    .stream()
    .map(player -> player.getName())
    .collect(Collectors.toList());
playerNames.forEach(System.out::println);

Groovy:

println players.collect{it.name}

Comments

• A stream is needed to be created first before executing the Lambda.  Then the collect() method is invoked on the Stream - this is needed to convert it back to a List. This makes code more verbose. 

• That said, if all you are doing is printing the list, you can just do...

  players.stream()
     .map(player -> player.getName())
     .forEach(System.out::println);
  

 

Perform a Reduction calculation

Java

System.out.println("Max player rating only...");
Optional<Integer> maxRatingOptional = players.
   stream()
   .map(RugbyPlayer::getRating)
   .reduce(Integer::max);
String maxRating = maxRatingOptional.isPresent() ? maxRatingOptional.get().toString() : "No max";
System.out.println("Max rating=" + maxRating);

Groovy

def here = players.inject(null){ 
    max, it -> 
        it.rating > max?.rating ? it : max
} 

Comments

• In the Java version, the reduce operation is invoked on the Stream.  There are three different versions of this method.   In this version, no initial value is specified meaning and an Optional type is returned.  The input parameter of type BinaryOperator.  Because BinaryOperator is a functional interface it means a lamda expression or method reference can be used to specify its value.  In this case, the method reference Integer.max() is used.
• The null safe operator is used in the Groovy inject closure - so that the first comparsion will work 
• In Java, it is possible to avoid the isPresent check on the optional by just doing...

  players.stream()
     .map(RugbyPlayer::getRating)
     .reduce(Integer::max);
     .map(Objects::toString).orElse("No max")
  

Summary

• Groovy is still far more terse
• However, some of the function operations in Java are lazily run.  For example map(), filter() which are considered intermediate.  Intermediate operations produce antoher Stream.  They won't execute unless a terminal function e.g. forEach, collect, reduce is invoked on the stream.  Terminal functions are value or side-effect producing. 
• Intermediate operations can either be stateless or stateful.  Stateless operations like map() or filter() can operate on elements independently. Stateful operations like distinct() or sorted() may incorporate data from previously seen elements.  
• The elements of a stream are only visited once during the life of a stream. Like an Iterator, a new stream must be generated to revisit the same elements of the source.
• In Java processing streams lazily allows for two performances efficiencies:
• fusing of multiple operations to minimise passes of the data
• avoiding examination of the data unless it is necessary.  A stream() may have an infinite number of elements, but a findFirst() or limit() operation will mean that only data that needs to checked will be.
• This may the code more verbose in cases, but it also means it can be more performant.
• Groovy also offers some lazy functions. 

Full Java code here. Full Groovy code here.

Scala Syntax: 7 points

A few years back I dipped into some Scala as a hobby language. Recently, in order to get a quick overview of Spark I did the 'Big Data Analysis with Scala and Spark' from Coursera. It's a great course. But, one aspect I found challenging was just getting my head around Scala syntax again. Some of it, yeah the basic stuff can be counter-intuitive depending on your perspective.

1. Method / Function Definition

Typing on the right rather than the left. Consider this simple function definition:

def sayHello(param: String): String = {
    "Hello" + param
}

Javaholics will note:

• The return is specified at the end of the method definition, rather than the beginning. 
• The type of the parameter is specified after the parameter name rather than before. 
• Before the function body there is a = 
• There are two colons (:), one between the parameter and the type and one before the return type.

2.  Unit

Google "Unit" and you be quickly told you that Unit is the Scala's version of Java void.  But, Java’s void is a keyword.  Scala’s Unit is a final class which only has one value: () - which is like an alias for no information. Unit indicates a method returns nothing and therefore has side effects, something we don't want to do much of in Scala. So is that counter intuitive? No.
But here is what I find is. If a function has no return type in the function definition and no equals it means Unit is implicitly the return type. Example:

def procedure {
    println "This String is not returned"
}

procedure: ()Unit

Big deal? Of course not. But what about:

def procedure {
     "This String is not returned"
}

Expect the String to be returned, it wont be. How about this?

def addNumbers(a: Integer, b: Integer) {
    return a + b
}

This will give a compile warning: :12: warning: enclosing method addNumbers has result type Unit: return value discarded return a + b It will compile but nothing will be returned:

def addNumbers(a: Integer, b: Integer) {
    a + b
}

will give no compile warning and will also return nothing.

3.  Underscore

In anonymous Scala functions, _ is like Groovy's it. In Groovy we can to multiple all numbers between 1 and 5 we can do:

(1..5).collect {it * 2}

In Scala we can do:

(1 to 5).map{_*2}

However, in Scala, the second time _ is referenced, it refers to the second parameter

val ns = List(1, 2, 3, 4)
val s0 = ns.foldLeft (0) (_+_) //10

4. Passing anonymous functions. 

Pass one anonymous function and you don't need any curly parenthesis. Pass two and you do.

def compose(g:R=>R, h:R=>R) = (x:R) => g(h(x)) 
val f = compose({_*2}, {_-1})

5. Arity-0 

When a method has no arguments, (arity-0), the parentheses can be omitted in invocation

size()
...
size  // do it like this 

But this technique should never be used when method has side effects. So,

queue.size // ok
println // not ok do println()

6. Declare parameter types

Function defiinitions / Method definition have to declare parameter types but function literals don’t.

def addNumbers(a, b): Number {
:1: error: ':' expected but ',' found.

7. Ternary Operator

There is no ternary operator in Scala. There is one in Java, Groovy, JavaScript. Python 2.5 added support for it. Instead you can do if else on one line and since if / else is an expression you can return a value. For example: In Java we would do:

(eurovision.winner == "Ireland") ? "Yippee" : "It's a fix"

Scala, it's:

if (eurovision.winner == "Ireland") "Yippee" else "It's a fix"

And some more REST tips

In previous blog posts I have covered some ideas and tips for achieving a REST architecture. In this post, I cover a few more ideas and tips.

Caching

• Caching is a big part of the original dissertation.  See section 5.1.4 
• Strategies include validation (client checks it has the latest version) and expiration (client assumes it has the latest version until a specified time)
• Expiration:
• Expires header tells client when resource is going to expire. The value 0 means avoid caching
• Cache-Control
• Use max-age directive to specify how long response should be considered valid for; s-maxage for shared caches
• Can also be used in requests no-cache means re validate response with server
• Validation
• Etag - unique version of resource. Used in conjunction with If-none-match request header
• Last-Modified - tells client when resource last changed

Controller APIs

• When something does fit neatly to a CRUD operation, consider a Controller API

Handling Dates

• Use ISO-8601 for your dates - better for natural sorting, handles timezone, locale nuetral, support from most programming languages
• Accept any timezone as anyone in the world may call your API
• Store in UTC, not in your server's timezone.  There should be no offset when persisted.
• Return in UTC.  Allow the client to adjust to its timezone as necessary
• Don't use time if you don't need it.  If Date only suffices, only persist Date. This means, timezone complexity goes away. 

HEAD

• HEAD action should return response headers

Headers

• Always return what headers are useful.  Consider: 
• Content-Type
• Content-Length
• Last-Modified
• ETag
• Location

Hypermedia (advantages)

• Less coupling
• Consistent format for links => cleaner client code
• Developer productivity: API's easier to navigate 
• Make easier to introduce services in a more granular way
• Code easier to debug - messages always have the URL that created them via the self link

Hypermedia (choices)

• HAL - reduces Address coupling 
• SIREN - reduces Address and Actions coupling
• Collection+JSON (CJ) - reduces Address, Action and Object coupling

Idempotent

• Can be called several times and return the same result
• OPTIONS, GET, HEAD, PUT and DELETE are all idempotent

Long Running Requests

• Some operations take a long time.  In such cases, consider returning a 202 with the location field set to a URL the client can poll to check for operation progress.

Method not allowed

• If an API only supports GET, it should return a 405 for any PUT, POST, DELETEs etc

Must Ignore Principle

• Clients should ignore data they are not interested in. This makes it much easier for APIs to be backwardly compatible .  If an API returns extra data and some clients aren't expecting it they will just ignore it. 

Not acceptable

• When a resource doesn't support a specific media type, it should return 406  (see Masse, Rule: 406 (“Not Acceptable”) must be used when the requested media type cannot be served

OPTIONS

• OPTIONS should return what actions are available on a resource

Partial Update

• Handle partial updates with  PATCH

Query

• The query component of a URI should be used to filter collections

Resource Creation

• When a Resource has been successfully created a 201 should be returned 
• The location header should indicate the URL to get the Resource. 

Safe

• Actions are considered Safe if they Do not modify resources
• OPTIONS, GET and HEAD are safe

Self link

• Response bodies should always include a self link - the URL that was used to return the resource. 

Singular or Plural?

• Use Singular for Singular Document type resource  - when there can only be one.  For example: /humans/12343343/head
• Otherwise plural

REST: Using a Controller endpoint?

In REST architectures, the fundamental concept is a Resource.   A Resource represents anything that’s important enough to be referenced as a thing in itself.   For example, a Shopping Cart, a Book or a Car.  The next fundamental concept is the Uniform Interface for accessing and manipulating the Resources.  In HTTP land usually means:

• Create is POST 
• Read is GET 
• Update is PUT (or PATCH for Partial Update) 
• Delete is DELETE

There are of course other concepts (statelessness, caching etc) but for this blog post, let's just focus on Resources.

In the real world,  many things map nicely to Resources.  However, inevitably somethings won't map so nicely to resources. This is usually a minority of operations for example reset password. It's possible to model these as either

•  a PUT on /password/ 

or as

•  a Controller endpoint and a POST to /resetpassword 

The latter may be considered to be closer to programmatic REST than pure REST, but there are times when clients and customers will want you to be pragmatic. This article gives suggestions regarding when to consider using the Controller option.

Does the action Map to a CRUD? 

Several actions in a real world application will not map nicely to a Create Read Update Delete (CRUD). For example, Paypal's cancel billing agreement API is:

POST /v1/payments/billing-agreements/agreement_id/cancel

The cancel action rarely maps nicely to a CRUD for a resource. It could be interpreted as:

• some resource gets be created (A cancel record) 
• some resource gets updated (some status column could be getting set to cancelled) 
• or some resource gets deleted (a order request gets deleted). 

Why should the client have to care about how cancel is handled?  Couldn't it always change? In some case API's have got around the doesn't map nicely to a CRUD problem using HTTP tunneling. For cancelling a billing agreement this would like:

POST /v1/payments/billing-agreements/agreement_id

with body:

{
  "operation":"cancel"
}

This is considered an anti-pattern and should never be used. Instead a Controller end point should be used.

 

Resource State or Workflow? 

In a REST architecture, every request between Client or Server will usually change a Resource State (write operation) or the Application State (a query or read operation). However, in the real world workflows are inevitable. For example, a reset password flow usually consists of:

• Asking the user for the userId (usually email) 
• System checking that email exists on the system 
• Sending the user an email with a link to reset the password 
• Ensuring the user only has a set amount of time to click the link 
• When the user clicks the link they may be asked a bunch of questions 
• They will be asked to retype their new password to ensure there's no typos 

When an client action is part of a complex workflow, Resource state and Application state changes may not be easy to model. They may not happen synchronously and they could change based on how the workflow is modelled or when the workflow needs to add an extra step. In such scenarios, consider using a Controller end point.

 

REST without PUT 

In some situations, arguments can be made for avoiding PUT and instead using POST to a different endpoint which signifies intent. For example, to change address instead of invoking a PUT to /address/, the client would invoke a POST to /changeaddress and avoid PUTs altogether.  One example where this approach is useful is when handling asynchronous operations and you are trying to make clear atomic consistent operation.  So for example, if changing address takes a long time and you would rather return a 202, with a location field for the client to poll, if you use the /changeaddress you can then leave /address endpoints as those that are only atomically consistent.

So, any PUT or POST to address, means if you were to immediately do a GET you would get the consistent view of the Resource.  This approach is also useful if you want to model the Business event rather than the actual resource that is changing.  So for example, suppose 6 or 7 things need to take place when a Bank account has been closed for a Business process perspective.  All on the back end in the same thread / transaction.  Again, here POST to controller endpoint such as /accountclosed makes more sense then /DELETE to /account.

See this article for more info. 

Summary

So why there may be subjectivity involved on when to use a controller style endpoint.  The above may at least help to you to make a decision.  Remember, it should always only be a minority of APIs where you consider this approach. You are outside the conventional Uniform Interface for unique style operations but you want to still make them feel intuitive to clients of the API. 

Testing your code with Spock

Spock is a testing and specification framework for Java and Groovy applications.  Spock is:

• extremely expressive 
• facilitates the Given / When / Then syntax for your tests 
• compatible with most IDEs and CI Servers.

Sounds interesting? Well you can start playing with Spock very quickly by paying a quick visit to the Spock web console.  When you have a little test you like, you can publish it like I did for this little Hello World test.

This Hello World test serves as a gentle introduction to some of the features of Spock.

Firstly, Spock tests are written in Groovy.  That means, some boiler plate code that you have with Java goes away.  There is

• No need to indicate the class is Public as it is by default.
• No need to declare firstWord and lastWord as Strings 
• No need to hurt your little finger with a ; at the end every line
• No need to explicitly invoke assert, as every line of code in the expect block gets that automatically.  Just make sure the lines in the then: block evaluate to a boolean expression.  If it is true the test passes otherwise it fails.  So in this case, it is just an equality expression which will either be true or false. You can have as many expressions as you want.

So less boiler plate code what next?  Well you know those really long test names you get with JUnit tests, well instead of having to call this test, helloWorldIntroductionToSpockTest() which is difficult to read, you can just use a String with spaces to name the test: Hello World introduction to Spock test. This makes things much more readable.

Thirdly, the Given: When: Then: syntax,  enforces test structure.  No random asserts all the test.  They are in a designated place.   More  complex tests, can use this structure to achieve BDD and ATDD.

Fourthly, if I were to make a small change to the test and change the assertion to also include Tony,  the test will of course fail. But when I get a failure in Spock, I get the full context of the expression that is tested.  I see the value of everything in the expression.  This makes it much quicker to diagnose problems when tests fail.

Not bad for an introduction.  Let's now have a look at more features. 

Mocking and Stubbing

Mocking and Stubbing are much more powerful than what is possible with JUnit (and various add on's). But, it is not only super powerful in Spock, it is also very terse, keeping your test code very neat and easy to read.

Suppose we want to Stub a class called PaymentCalculator in our test, more specifically one of its method, calculate(Product product, Integer count).   In the stubbed version we want to return the count multiplied by 10 irrespective of the value of product.   In Spock we achieve this by:

PaymentCalculator paymentCalculator = Stub(PaymentCalculator)
paymentCalculator.calculate(_, _) >> {p, c -> c * 10}

If you haven't realised how short and neat this is, well then get yourself a coffee.  If you have realised well you can still have a coffer but consider these points:

1. The underscores in the calculate mean for all values 
2. On the right hand side, of the second line, we see a Groovy Closure. For now, think of this as an anonymous method with two inputs. p for the product, c for count. We don't have to type them. That's just more boiler plate code gone. 
3. The closure will always return the count time 10.  We don't need a return statement.  The value of the last expression is always returned. Again, this means less boiler plate code.  When stubbing becomes this easy and neat, it means you can really focus on the test - cool. 

Parameterised Tests

The best way to explain this is by example.

@Unroll
def "Check that the rugby player #player who has Irish status #isIrish plays for Ireland"(String player, Boolean isIrish) {
    given:"An instance of Rugby player validator"
    RugbyPlayerValidator rugbyPlayerValidator = new RugbyPlayerValidator()

    expect:
    rugbyPlayerValidator.isIrish(player)  == isIrish

    where:
    player               ||  isIrish
    "Johny Sexton"       ||  true
    "Stuart Hogg"        ||  false
    "Conor Murray"       ||  true
    "George North"       ||  false
    "Jack Nowell"        ||  true

}

In this parameterised test we see the following:

1. The test is parameterised. The test signature having parameters tells use this, as do the where block.  
2. There is one input parameter player and one output parameter - which corresponds to an expected value. 
3. The test is parameterised five times.  The input parameters are on the left, output on the right. It is, of course, possible to have more of either, in this test we just have one of each. 
4. The @Unroll annotation will mean that if the test fails, the values of all parameters will be outputted. The message will substitute the details of player into #player and the details of the Irish status substituted into #isIrish. So for example, "Checks that the rugby player Jack Nowell who has Irish status true plays for Ireland"

Again, this makes it much quicker to narrow in on bugs. Is the test wrong or is the code wrong? That becomes a question that can be answered faster.  In this case, of course it is the test that is wrong.

All the benefits of Groovy

What else? Well another major benefit is all the benefits of Groovy.  For example, if you are testing an API that returns JSON or XML, Groovy is brilliant for parsing XML and JSON. Suppose we have an API that returns information about sports players in XML format. The format varies, but only slightly, depending on the sport they play:

 
  
    Joey Carberry
    Teddy Thomas
  




 
   
     Lionel Messi
     Cristiano Ronaldo
   
 

We want to just invoke this API and then parse out the players irrespective of the sport. We can parse this polymorphically very simply in Groovy.

def rootNode = new XmlSlurper().parseText(xml)
def players = rootNode.'*'.Players.Player*.text()

Some key points:

1. The power of dynamic typing is immediate. The expression can be dynamically invoked on the rootNode. No verbose, complex XPath expression needed.
2. The '*', is like a wildcard. That will cover both RugbySummaryCategory and FootballSummaryCategory.
3. The Player*, means for all Player elements. So no silly verbose for loop needed here 
4. The text() expression just pulls out the values of the text between the respective Player elements. So why now have a list all players and can simple do:players.size() == 4. Remember, there is no need for the assert. 

Suppose we want to check the players names. Well in this case we don't care about order, so make more sense to convert the list to a Set and then check. Simple.

players as Set == ["Joey Carberry", "Teddy Thomas", "Lionel Messi", Cristiano Ranaldo"] as Set

This will convert both list to a Set which means then order checking is gone and it is just a Set comparison. There's a tonne more Groovy features we can take advantage of. But the beauty is, we don't actually have to. All Java code is also valid in a Groovy class. The same hold trues for Spock. This means there is no steep learner curve for anyone from a Java background. They can code pure Java and then get some Groovy tips from code reviews etc.

Powerful annotations

Spock also has a range of powerful annotations for your tests. Again, we see the power of Groovy here as we can pass a closure to these annotations. For example:

@IgnoreIf({System.getProperty("os.name").contains("windows")})
def "I'll run anywhere except windows"() {...}

Or just make your test fail if they take too long to execute

@Timeout(value = 100, unit=TimeUnit.MILLISECONDS)
def "I better be quick"() {...}

So in summary Spock versus vanilla JUnit has the following advantages:

1. Test Structure enforced. No more random asserts. Assertions can only be in designated parts of the code. 
2. Test code is much more readable. 
3. Much more information on the context of the failed test
4. Can mock and stub with much less code
5. Can leverage a pile of Groovy features to make code much less verbose
6. Very powerful test parameterisation which can be done very neatly
7. A range of powerful annotations. 

And one of the often forgotten points is that your project doesn't have to be written in Groovy. You can keep it all in Java and leverage the static typing of Java for your production code and use the power and speed of Groovy for your test code.

Until the next time take care of yourselves.

When a REST Resource should get its own Address?

Background 

Author's note

In a purist REST approach, all endpoints (except the starting endpoint) are opaque and their various details shouldn't need to be published.  Even if this approach is being used, the points in this article are relevant as Server logic will have to determine when something requires a end point or not. 

Introduction 

In a REST architecture an entity or a resource (for the rest of the article the term entity will be used) may or may not have its own address.  For example, suppose we have an inventory application merchants use to sell their products. Immediately it is possible to see a Product entity.  It's URL will look something like: /product/{id}

Now, it is possible for the merchant selling the Products to add his / her own comments to the Products.  For example, "Sells very well on Fridays" or "Consider changing price if product doesn't start selling".   A Product can have 0..* Comments.  As stated, the Product has its own address: /product/{id} for example /product/1231233

and a response payload like this

{

    "id":"1231233",

    "type":"Beer",

    "comments": [{

             "id":"1",

             "comment":"Sells very well on Fridays"                 

     }, {

             "id":"2",

             "comment":"Consider changing price if product doesn't start selling"  

     }]

}

As can be seen, the payload returns a collection of Comment Objects. Should the individual comments each have their own address as well or is it okay that they are just embedded into the Product response? To help the answer this question the following should be considered.

Does the Entity have any meaning outside the Containing Entity Context? 

If an Entity (for example Comment) has meaning outside their containing Entity (for example Product) then they should have their own address.  For example, suppose the Entity was Student and the Student returned a list of Universities he / she had studied.   These Universities have their own meaning outside the Student. So obviously the University should have its own address. In the Activity / Comments scenario, the Comments only exist for the activity.  No other Entity will ever reference them or need to reference them. Therefore further aspects needs to be considered.

Is it desirable to perform actions on the individual entities? 

Should the client be allowed to create, read, update or delete the individual entity? These have to be considered separately.

Writes: Create, Update, Delete 

In the Product / Comments scenario, a Comment would never be created outside or without an Product. It is essentially added to an Product. This could be considered as a partial update to the Product.  However, an update or delete to an existing Comment could also be considered a partial update to the Product.   This creates complexity on how to differentiate between Create / Updates and Deletes of a Comment using a partial update on the Product.  If this is required, it would be much simpler to create a contextual address for the Comment (which indicates the hierarchical nature of the Product / Comment) then allow the Client sent POST, PUT, PATCH, DELETES to that.

Example URL: /product/1231233/comment/1

Reads 

In some scenarios the parent containing Entity may not return all the information about the child Entities. For example, again consider the Product --> Comment scenario. Suppose the comment was very large. This would mean the payload for the Product was also very large.  In such cases, it might be more prudent for the Product to just return a summary of the Comment and if the client wants the full Entity to make an individual request.  Similarly, if there's a big performance cost to get an individual Entity (for example a 3rd party API has to be invoked to get all the information about the comment), it can make more sense to just send a URL link to the Entity rather the than the actual entity contents.

N+1 Problem 

If individual Reads are required, be careful that the N+1 problem doesn't then get introduced. For example, suppose a Product could have 100 Comments. the Product API will only return a summary of the Comment and a link to each individual comment if the client wants all the information. However, if the client wants every single comment, this means there will now be 100 HTTP requests. If this is a potential scenario, then a secondary endpoint which aggregates all the comments into the Product should be considered. This is similar to the API Gateway pattern.

Surface Area of Endpoints

In any architecture when contracts are published, if there are too many it can become very unwieldy for developers to understand. Most well known APIs (e.g. PayPal, Amazon, Twitter, Google) usually only have about 20 - 30 addresses. This is a good aim to have. If there are 5,000 different addresses it can become way too large and difficult to control etc.

In summary, the decision diagram provides guidance on what you should do.

Are you forgetting your Agile values?

A while back I wrote why sometimes Agile will fail.  In this post, I will focus on the specific misunderstandings of Agile values.   When people ask if you're Agile, they basically think:

• Do you have stand ups?
• Do you have retrospectives?
• Do you have stories?  
• Do you use yellow post its?
• etc

Such ideas belong to an Agile process called Scrum which is the most popular Agile process but isn't the only one.   There are other processes: Kanban, RUP, XP, etc...  You don't necessarily have to be Scrum.

For me the most important thing that came from Agile was the actual manifesto.  It should be read by everyone working in software at the beginning of every year and at the beginning of every project. Then it should be discussed with team members and people should be encouraged to give specific examples they have of the values and principles detailed in the manifesto.    In this blog post,  we'll have a look at the values.

8 Values with emphasis on 4

There are four points which detail 8 values in the manifesto

1. Individuals and interactions over processes and tools
2. Working software over comprehensive documentation
3. Customer collaboration over contract negotiation
4. Responding to change over following a plan

Then there is the very important sentence: "That is, while there is value on items on the right, we value items on the left more."  It is worth saying that sentence twice.  Maybe three times... four times... whatever it takes so you never forget it.  Why? Well let's take them one by one.
 

Individuals and interactions over processes and tools

So does this mean if you are Agile do you stop worrying about process?  No.  No project can ever be successful without process.   From an Agile perspective, it means, you value process. But, you value Individuals and interactions more.   So for example,  say you spend lots of man hours in a process that isn't really adding value to the project or customer. When you look at this critically it is because developers and testers don't talk to each other.  Instead they have a convoluted process so they think they can blame each other when a production defect happens.  Lots of man hours goes into this.  But, it would be much more efficient if they talked regularly to each other which then negates the aspects of the process which are making it convoluted.  So the challenge is two fold:

• Ensure people talk to each other regularly
• Ensure your processes are efficient 

So for example, it makes more sense to have a regular show and tell then no interaction whatsoever and a massive delivery at the end where you are hoping you will get customer satisfaction because of some long complex process.   Aim to make individuals, interactions in the heart of your processes.

Working software over comprehensive documentation

This is a classic misunderstanding.  Somebody wants comprehensive documentation for some complex functionality and a developer retorts: "Hey Dinasour, this isn't waterfall, there is no need for detailed documentation".   Wrong.  Documentation is still required. The point here is that with Agile you shouldn't be spending massive amounts of man hours on documentation when your software is riddled with bugs.   It is more important that your software works.  This means more time should be spend on excellent automated tests that have sufficient functional coverage.  This isn't always easy to do but you should ensure this is done. 

Customer collaboration over contract negotiation

Thirdly, do you stop doing contracts agreements with customers on your projects.  No. The point here is you spend more man hours with meetings collaborating with customers than you do teasing out a contract with lawyers.  Instead of spending a massive amount of time to get sign off on a massive project, you should collaborate thru the life cycle of the project, breaking it up into small increments, take on board feedback and work together towards a common goal: project success. 

Responding to change over following a plan

Lastly, do you stop planning?  Of course not.  But again, what is the point planning down to the nitty gritty if you cannot even respond to change? Could you imagine a customer asked for a tiny change to how a UI was displaying data and the developer team respond with: "Sorry that wasn't in our 6 month plan"? It is paramount that architecture facilitates reasonable change and if it can't the project will struggle to really embrace an agile philosophy. 

Summary

Agile is actually about putting more constraints on your software methodology.    As an analogy, the REST architecture style puts constraints on your architecture for example the Uniform Interface, Statelessness and Caching capabilities.  The idea is then by sticking to these constraints (and that's a technical challenge) you get benefits.  In the case of REST, your architecture will be more scalable, extensible and lead to much higher developer productivity for API consumers.  In the case of Agile, by sticking to the constraints of valuing 8 key concepts but putting even more emphasis on 4, your project will have greater chance of success.