00001 : the what and the why?

00001 : the what and the why?

I've been toying with calling this post:

ServiceStack, "talk DTO to me ;)"

...but will perhaps save that one for a t-shirt, it tickles the inner geek!

My team has recently released a number of plugins geared towards microservices for ServiceStack and have more on the way. So now is perhaps the right time to provide some context into what we are doing and why.

This is part one of a series. This first part covers the background and rationale behind the decision to explore microservices and in the subsequent parts, we cover the specifics of tools, technologies and code to make it happen.

The posts are quite long and information-dense, but I fear if I unpacked all that information, it would end up as a book, which I don't have the time to write - too busy coding!

If you are interested in ServiceStack, or even just the practicalities of implementing microservices, I hope this series is of some interest or help to you, dear reader.


ServiceStack, a journey into the madness of microservices
  1. Context: the what and the why?
  2. Debugging and logging
  3. Service discovery, load balancing and routing
  4. Service health, metrics and performance
  5. Configuration
  6. Documentation
  7. Versioning
  8. Security and access control
  9. Idempotency
  10. Fault-tolerance, Cascading failures
  11. Eventual consistency
  12. Caching
  13. Rate-limiting
  14. Deployment, provisioning and scaling
  15. Backups and Disaster Recovery
  16. Services Design
  17. Epilogue

Uh oh!

Now first the elephant in the room: we have used the term "microservices" which, I fear has already become a dirty word in software development from overuse.

If I hear micro services one more time - Kelly Sommers

Suffice to say the gif wasn't of kittens smooching!
I'm sorry, Kelly, this post is full of them.

History is often not kind to over-hyped terms like microservices and it may end up being obscured by failed projects, poor implementations or the same vendor marketing techno-babble that SOA suffered before it is proclaimed to be dead, and Self-Contained Systems or Serverless Architectures are the new hotness!

I've read and heard many times over the past few years variations of "microservices is just describing what I always thought SOA was" and I think they're right, it is easy for these terms to become so overused that fatigue sets in and there is a backlash.

Nothing about microservices or SOA is new to me, they are just concepts that sound like the answer to every problem.

Instead of writing one large complex thing, write smaller simpler things, that do one thing well and compose them together

Both OO Composition and Functional Composition describe this approach. In fact, this idea is found countless times in other places like the UNIX philosophy and the actor model both from 1973 and have their roots even further back than that. I'll leave that exercise for the historians among you.

The take-away here is that these concepts are not new at all, so regardless of the term used, they are trying to convey concepts of composability.

Yet, I think that microservices really isn't the solution for most problems. If you have a problem and use microservices to solve it, you now have 21 problems and counting!

This is why the advice you hear often is to build a monolith first.

Then, once it matures, and you understand the problem domain and its bounded contexts, should you consider making it a distributed system by breaking it apart into smaller 'micro' services.

So what's the plan, Stan?

So broadly this post is about 'things you need to think about when building microservices', but more specifically it is about building them with ServiceStack.

I've been using ServiceStack for years and it's been running a bunch of internal business-critical API's for us, I'm a huge fan of the DTO-first approach that is at the heart of ServiceStack's design philosophy.

So for those who aren't familiar with ServiceStack, here is the over-simplistic introduction.

The premise is simple to grasp but deceptively powerful.

Your service methods take in a single parameter, the Request Data Transfer Object (DTO); which is just an implementation-free Plain Old CLR Object (POCO) class.

These DTOs are what is used to define your Services Typed API Contract.

So DTOs are the means by which to send data to your services which optionally receive DTO data in return.

Once you accept that simple but fundamental premise and start to build API's, you will discover that it is amazingly flexible.

This is messaging.

A simple DTO can be routed to the right method depending on the HTTP Verb it is sent with.

You can send and receive XML DTOs in one app and JSON in another. Need to switch to CSV or MessagePack? No problem, it can do that too. How about asynchronously? No problem, supported out the box.

Want to send a batch of DTOs at once? Again it will just work.

This is just the tip of the iceberg really, the DTO-first approach abstracts away details like formats, transports, persistence, caching and much, much more.

I've battle-tested it, I understand where it's coming from and it has proved to be an excellent replacement for everything web-related that was previously prefixed "MS" and I couldn't be more happy and comfortable with that decision.

ServiceStack however, is really only the service part. It doesn't include many of the pieces required to glue distributed systems together. That isn't a failing in any way on its part, its simply not where it's coming from.

So, why are we trying to do mean things like implement microservices with it?

You see, creating services in ServiceStack is easy. Once you get used to the DTO-first coarse-grained style, you find that your services are easy to extend, update, bend, reconfigure and generally cater for your changing needs, but your service logic doesn't have to be rewritten to support this.

Perhaps you need to experience the pain of refactoring and rolling out brittle or chatty RPC contract-based services that break consuming clients (WCF, SOAP and WebAPI, I'm looking at you!) before this approach just clicks and you finally "get it". Once you do, you're unlikely to want to go back.

Another thing I have found over the years is that API's tend to outlive applications by an order of magnitude. The well-designed API is still serving data long after the apps that it was built for are long gone.

But I digress. We are exploring microservices for two primary reasons, code complexity and scale.

Complexity

We have a mature set of monolith apps and services that operate well, but, they have shared data sources and some shared dependencies in the form of Nuget packaged libraries.

It is difficult to maintain good separation of concerns with the shared libraries and shared data sources. Over time boundaries begin to become blurred.

These are very real pain points in our development.

We have also reached a point where system complexity makes it challenging to reason about the current state within the system, and lastly when releasing updates, we have issues with lock-step.

Moving toward a coarse-grained set of loosely coupled API services with isolated datastores and some event-sourcing is where I think we can begin to rein in these problems.

We have the following posted on our internal wiki design-notes to provide us with a constant reminder of the challenge we are facing in choosing this path.

Monolith to Microservices

Scale

Now scale can mean different things. In this context I am not referring to traffic scaling but of scaling development. To enable distributed development teams to collaborate on a message-driven system, where communication and development is achieved through DTO-driven contracts.

This is where ServiceStack comes in. I personally see ServiceStack as a transport-agnostic and format-agnostic messaging platform than a web services framework.

Oh dear, that sounds like a lot of marketing techno-babble!

I'm not saying it isn't great for serving websites and restful apis, but that isn't the focus of this post.

To try and explain then, ServiceStack might not support a nanomsg, zeroMQ or Redis-like wire protocol at the moment, but its architecture means it could and your core business-services wouldn't have to change a single line of code. Not one line!

One example of how you can layer new functionality on top of ServiceStack's architecture is our prototype event-sourcing and CQRS pattern delivered as a single nuget plugin. It has very little API surface-area of it's own, using instead what is already available in ServiceStack.

If you haven't already checked it out, it is well worth a look. It is, I think, a fantastic implementation by David Brower that distills a lot of the complexity of event-sourcing into a really natural DTO-first experience and makes it very easy to use in the right way. I'll discuss how this fits in later.

So the same exact same code to consume and produce http calls, can also subscribe to messages from a MQ and subscribe to events from eventstore streams. That for me is practically black magic and it's really compelling to build systems on.

With the awesome Demis Bellot's help, we have also been able to get a really nice service gateway pattern in ServiceStack from v4.0.56 onwards, and although we didn't tell him at the time, (shhhhh) but it is what we were hoping for as it's the gateway to everything else (pardon the pun), so we were delighted when it was added to ServiceStack.

It opens up many possibilities for where we are headed.

So where the heck are you heading?

Breaking apart systems (Distributing Systems), which is really what microservices is trying to do, introduces a lot of challenges, which, instead of eliminating complexity in your code, shifts it into infrastructure and operational complexity.

It should be a conscious choice as to whether you think this trade-off is worth making as you will encounter things like the fallacies of distributed computing and these are often non-trivial problems. You need solid foundations on which to build, and everything is harder when you have networks sitting between your services.

Magically solving these problems for you is where the vendors will pounce, offering single-source solutions or spout nonsense terminology that makes me chuckle, but, there are people out there who hear words like hyperconverged and get so excited they just start vomiting money. Get your buzzword bingo card's out cos we all gonna be winners, the author of that piece really seems to like using it!

But it raises an important point, why not use an existing Enterprise Service Bus (ESB) type solution?

If that is the right fit for your circumstances, then you absolutely should, but for me personally, perhaps I've been burned too many times by vendors becoming disinterested or rewriting the integration points every few years chasing new business, that the prospect of coupling my systems to them is unappealing.

It isn't a co-incidence that I have bug-free, dependable services running in production that I wrote ten years ago. They have no coupling to vendor specific solutions, yet all the integration middleware I wrote for 2007 versions were trashed in 2010 and again for 2013. I'll leave it for you to guess who that was for.

The system you write is the system you understand, debug, maintain and control.

Ultimately, I want systems where the infrastructure is composed, just like my services are.

When something better in any given area of infrastructure comes along, I want to be able to plug it in, without rewriting that whole system or trashing my business logic in 'the big rewrite'.

These for me, are core business systems that have lifespans similar to programming languages lasting decades if designed and maintained well, long after the vendor solutions are gone or you are forced through multiple painful upgrades because OS upgrades don't support the older versions.

No really, where ARE you heading?

Ok, so to recap , microservices are bad, really hard, not the solution; OK, you just don't do them...

..

We're doing microservices :)

... or at least a form of them that makes sense to us and doesn't stray far from the DTO-first ServiceStack philosophy.

If we want to fall into the pit of success though, there are a lot of things we need to get right. The glue that binds it all together into a coherent whole is hard to get right and involves a long list of non-trivial things you need to consider:

  • Distributed debugging and logging
  • Service discovery, load balancing and routing
  • Service health, metrics and performance
  • Configuration
  • Documentation
  • Versioning
  • Security and access control
  • Idempotency
  • Fault-tolerance, Cascading failures
  • Eventual consistency
  • Caching
  • Rate-limiting
  • Deployment, provisioning and scaling
  • Backups and Disaster Recovery
  • Security and access control (twice cos its really important)

There are probably a few missing from that list too, but the number of things required to make it all click is why there is no step-by-step guide to properly implementing microservices anywhere.

Every article, talk and presentation that discusses microservices touches on this complexity, but the variety of options means that it often ends up lot like 'drawing the owl'
Implementing microservices

Remember, don't do microservices.

Still with me?

In each part, we will cover a topic on the list in more depth, as well as discussing how we arrived at the design decisions we made, the tradeoffs we are making and our progress so far.

So, let's do microservices!

next up: Logging and Debugging

How to load debug symbols in Visual Studio during debugging

If you've been working in Visual Studio for any length of time then you'll probably be familiar with the following message.

No Symbols Loads

You've set a breakpoint in the code from a 3rd party dependency for which the debug symbol file .pdb has not been loaded.

How can you tell what symbols have been loaded? Easy, it's in the Modules windows in the debug menu.

Modules view

Once you launch the window, you can then view all the loaded modules and whether their symbols are loaded.

To load the symbols for your module, just right click on the module and select Load Symbols as below.

Load symbol

Logging ServiceStack requests with Seq

Logging ServiceStack requests with Seq

Following on from my last post where I used a serilog enricher for servicestack exceptions I wanted to log all requests in a ServiceStack AppHost to Seq.

Seq

If you don't know what seq is, it is described on their site as follows:

Seq is the easiest way for .NET developers to capture, search and integrate structured log events

Basically this means your apps can send rich log data to seq and have a really great experience around searching and filtering.

ServiceStack

ServiceStack provides this functionality in the RequestLogsFeature plugin, which by default creates an in-memory history of the last 1000 requests or optionally can be persisted using Redis.

If I can instead log these requests to Seq where all my apps log to, I can better track the flow of requests from applications to my API's.

With the RequestLogsFeature as my starting point I've created a plugin which logs requests to an instance of Seq. It does this by posting over HTTP/S to Seq.

The source is available on GitHub. You can also find it on nuget as ServiceStack.Seq.RequestLogsFeature.

Usage

In your AppHost add the plugin and specify your seq url and optional apiKey.

public override void Configure(Container container)  
{
  // Config examples
  Plugins.Add(new SeqRequestLogsFeature("http://localhost:5341", "seq-api-key"));
}

That's it. Simples!

You can optionally configure the same options the native request logger has, such as logging response bodies, session tracking etc and it shares all the same defaults.

Message template

Seq uses message templates to generate 'uniqueness' for types of requests.
For the RequestLogger I used the http://server/request/path ignoring querystrings to define my requestTypes on seq.

This hopefully provides a good way of filtering the logs by the server and service method called.

Logging levels

  • All 400-499 http status codes are logged as Warnings
  • All 500 > http status codes are logged as Errors
  • All other requests are logged as Info

Security

Unlike the native request logger, which uses AppHost roles, you control logging permission using api keys and read permissions to those logs via Seq users, Both local and active directory users are supported.

The other native logger defaults have been kept but if you are logging any potentially sensitive information, you should also consider configuring ssl on your Seq instance to ensure the seq post requests are encrypted.

Log Retention

Again unlike the native request logger, seq has it's own retention policies which give you fine control over which logs to keep and for how long.

Versions

I've created two different versions. For ServiceStack version 3.x, use the v3.x of the nuget package. For ServiceStack v4.x, use ... you guessed it, the v4.x version of the nuget package.

Bugs + Requests

Log any bugs and feature requests to github or submit a pull request if you're able.

Wrapping up

In one line of code in your AppHost you can now log all you api's requests to Seq.
Seq