Using Service Fabric Partitioning to Concurrently Receive From Event Hub Partitions


The classic tale is you have Azure EventHub with a number of partitions, you run your receiver process in a singleton Azure Web Jobs or a single instance Azure Worker Role cloud service both will use “Event Processor Host” to pump events into your process. The host externalize offset state storage to Azure Blob Storage (to survive processor crashes and restarts). This works fine yet it has the following problem. You can not scale it out easily. You have to create more separate cloud services or web jobs each handling some of your event hub partitions. Which will complicate your upgrade and deployment processes.

This post is about solving this problem using a different approach.

Service Fabric Azure Event Hub Listener

Azure Service Fabric partitions are somehow logical processes (for the purists I am talking about primary replicas). They can exist separably each on a server, or co exist on a node. depending on how big your cluster is and your placement approach.

Using this fundamental knowledge we can create a service that receives events from Azure Event Hub as the following:

  • If the number of Service Fabric Partitions is greater than Event Hub Partitions. Then each Service Fabric partition should receive from multiple Event Hub partitions at once.
  • If the number of Service Fabric Partitions is equal Event Hub Partitions then each Service Fabric partition should receive from one Event Hub Partition.

Because we have absolute control over how the above mapping is done, we can also add 1:1 mapping (irrespective of the number of partitions here and there). or even custom funky logic where some service fabric service partitions are doing the receive the rest are doing something else.

The second piece of the new tale is state. Service Fabric has it is own state via the use of stateful services I can save few network I/Os by using them to store the state – reliably – internally on the cluster via the use of Reliable Dictionary. The state management for our receiver is plug-able. As in you can replace it with whatever custom storage you want (one of the scenarios I had in mind is running the listener in a stateless service and use a another packing stateful service).

The third piece of the new tale is usage. Event Hub Processor is great but it requires a few interfaces to be implemented (because it is a general purpose component). Ours is highly specialized single purpose component. Hence we can simplify the entire process by just implementing one interface (that represents what happened to events and when to store the offset).

The last piece of our new tale is the entire code wrapped in Service Fabric ICommnuicationListener implementation allowing you to use just as any other Service Fabric listener.

The code for the above is on github here

till next time


On Application Routers/Gateways


The discussion below is about Routing & Application Gateways. It is about a component a wrote the code for and is published here along with the documentation on design & how to use. The below is ranting on how things in this component are made in a certain way.

“Application Gateways are hard to build, but should not be hard to use”

Why Build an application Gateway or a Router

First: Why Application Routers/Gateways?

Your application will have requests coming to your external (or even internal) end points. These requests are not meant to be fulfilled by the application that received the requests but rather via backend of some sort. Most large scale (and especially those that are built for hyper-scale) have this requirement. Take the following examples:

  1. A SaaS application where users’ requests are routed to specific tenants. These requests might be web pages, Web Sockets, REST API Calls or even TCP/IP packets.
  2. An application that distribute the loads/requests types among multiple backend systems.
  3. An application that went through massive transformation (take M&A for example scenarios) where external API have changed, you will need a gateway in the middle to upgrade or bridge the protocol without the need to disrupt your existing external clients.
  4. Application Zoning/Containment/Partitioning/Segregation, where connections from the external world are authenticated, authorized then terminated at the gateway. The gateway then establishes connections to the backend systems.
  5. A microservices application deployed on top of a cluster managed by a cluster management platform such as Apache Mesos or Azure Service Fabric will need a gateway. While some examples refer to using 2 tiers (Web UX deployed on all nodes routing to compute in the backend such as Azure Service Fabric’s WordCount ( This approach does not deal with situations where Web UX might be subject to routing per request (take a SaaS application, A/B Testing, or an application that provision mesos jobs or Azure Service Fabric App instances to scale to meet the load or isolation requirements).
  6. Specific platform requirements such as offering Azure Service Fabric Actor framework to external – to the cluster – applications (where Actor clients cannot be used).

And many others.

Is Routing a New Thing?

No, as a matter of fact every time you used a web server internal routing happens on the OS – typically by a kernel component – to route requests to different processes responsible for different address. At a higher level, frameworks such as ASP.NET Web API or WCF preform per request routing (dispatching in case of WCF) to the target controller/method (Service/Interface in case of WCF).

But This Is at the Platform Level, What about the Application Level Are This Requirement New? Is It a Cloud Thing?

No & No. Application gateways has been a thing since forever. I remember building an Http application gateway 1998. And there are 10+ years old products out that perform various parts of Application Gateway logic. The Cloud came in to fulfill “hyper-scale” requirements. Where the application itself can be provisioned multiple times to support load or isolation (hence the stress on routing).

The Problem With Routing

If your requirements are about from point A to point B and both are the same semantics (say Web API) with fairly static URL then you don’t need a custom gateway and I would strongly recommend looking for existing solutions such as Azure API Management ( Or build a single purpose gateway. The problem is most requirements comes in the following forms:

  • Routing in multitenant solutions

If request is on https://<some-tenante>.<hist>.com/crm route it to http://node:port/<some-tenante>/crmapp (node is backend server) and add custom authorization header, then get the response and put in the original downstream payload while adding a custom header.

  • Routing in multi version and or A/B testing scenarios

If request is on https://www.<host>.com/api/customer and authorization header claim contains usergroup = “default” or usertype != “admin” then route it to http://node:port/api-v2/customer else route to http://node:port/api /customer

  • Routing in microservices like environments

if request is on https://www.<host>.com/api/customer then resolve microservices address list and perform round robin load balancing between them however if the request type is Post or Put or Delete then route only to primary partition (in case of Service Fabric).

  • Routing in protocol bridges

If request came on Web Sockets address ws:// then route based to http://node:port/<some-tenante>/api/customer and set Http Method based on messagePayload.Type MT, MT = “add” then Method = Post, MT = “update” method = post etc.

Sounds complex enough? those are typical requirements for an application gateway.

The problem is scary enough but relatively easy to solve. If you split the representing the logic from the actual execution

Routing Logic, Simplifying the Complex

You can easily represent the logic by a linked list where each node represents a condition and/or logic and is only executed if the executed successfully. In my code I called nodes Matcher (not the sexiest name I know). Consider representing them as the following:


//pseudo code 

// If request on bing then route it as Get to &lt;a href=""&gt;;/a&gt; and add a custom header “CUSTOM_HEADER” with value “Hello, World!”;

var head = new SetMethod(“Get”)

head.chain(new SetAddress(“”), new AddHeader(“CUSTOM_HEADER, “Hello, World!”), new MatchAddress(“bing”, MatchType.MatchHostNameOnly);

// for more concrete samples and implementation check

The above code describes routing and processing logic in an easy to understand fashion, and more importantly an easy extend framework. You can extend to add whatever matcher types you want. You will end up with something that looks like this (all images below are from the repo):


But what about my ANDs and ORs? This can also be represented by matchers, consider the following:

// pseudo code 

// If request on bing and user type is “dev” then route it as Get to &lt;a href="http://www.,"&gt;http://www.,;/a&gt; else &lt;a href=""&gt;;/a&gt; and add a custom header “CUSTOM_HEADER” with value “Hello, World!”;

// If request on bing then route it as Get to &lt;a href=""&gt;;/a&gt; and add custom header “CUSTOM_HEADER” with value “Hello, World!”;

var head = new SetMethod(“Get”)

var msdn = IsUserType(“Dev”)
msdn.chain(new SetAddress(“”))

   new OrMatcher(
             new SetAddress(“”)
   new AddHeader(“CUSTOM_HEADER, “Hello, World!”),
   new MatchAddress(“bing”, MatchType.MatchHostNameOnly));

// for more concrete samples and implementation check

Because of this type of branching the matchers are represented in memory as a tree not just a linked list.



Because Matcher is a .NET type you can subclass it in new types of matching that suits your application (the code published here contains most of the common stuff) or You can extend the existing ones with new capabilities specific to your application.

Executing the logic becomes a matter of mechanics as described in the documentation here (the basic idea a “Routing Context” is pushed through the linked list).



I have chosen to use this space to describe what and why we need application routers / gateways (the what and how along with source code is published on GitHub). I have also chosen to cover just one aspect of the complexity that usually comes in building them. Check the documentation to get an idea about the rest of the problems that a typical application router/gateway have to solve and how they were solved.

till next time



On October & Azure Service Fabric IoT Sample

Azure Service Fabric public preview is out today. Check the announcement here over most of October I have been busy – as in you wouldn’t get my usual ranting – putting together a reference architecture and sample for IoT on top of Service Fabric the sample is published here
The sample in brief – quoting the docs – is:
“This sample is a reference architecture & implementation for hyper scale IoT event processing. The sample uses compute and storage on Azure Service Fabric and integrates with Azure Storage, Azure Power BI, and Event Hubs.”
Follow the link above for the sample. for the rest of the samples check be sure to check the “Party Clusters” 🙂
Till next time

Azure Service Fabric: Multiple Communication Listeners Per Service

So I was off for a week, then got elbow deep on one of the projects. Slowly recovering to normal work rhythm. Until the work I am doing is completed – expected by end of October, stay tuned for future announcements 🙂 – I thought I’d share something of interest.

I am working with a Service Fabric service, where each service is expected to have:

  1. 0..n number of event hubs listeners. The event hub number can change as in add, remove hubs etc. Needless to say the service doesn’t need to stop, start, reset or upgrade in case of event hub
  2. 1 – Only one – REST control endpoint, where you can interact with the service to do:
    • Add Hubs.
    • Remove Hubs.
    • Get # of processed messages and other relevant telemetry data.
    • Stop/Start/Pause processing.

Service Fabric allows you to create one listener per service (for purest folks, per replica). More details here ( In order to allow this to happen I created a composite listener, basically sets on top a list of listeners, listeners can be added, removed without affecting service lifecycles. The composite listener ensure that newly added listeners status matches the composite listener. As in if the composite listener has been opened-async then the newly added listeners will be opened-async-ed as well.

The composite listener can work with any class that implements ICommunicationListener interface (including those provided in the public samples).

I published it as a gist on github here:

Till next time @khnidk


On Service Fabric & Multiple Wcf Endpoints

Service Fabric allows you to listen to arbitrary endpoints (irrespective of the protocol, messaging framework etc..) by funneling everything through CreateCommunicationListener method which returns a listener (discussed at a great length here). The listener itself have basic life cycle control primitives exposed to to you as Initialize, OpenAsync, CloseAsync & Abort methods.

The included libraries (with Service Fabric) further extends into Wcf specific listening with 1:1:1 mapping (one service replica: one Wcf Host: one Wcf endpoint) which works well in a lot of the situations.

The problem happen when this mapping is not found (for example):

  1. Legacy services  that were built to have 1 host: multiple endpoints). and you need to migrate as is (some refer to inter contract shared state in Wcf world) .
  2. Legacy services that uses *weird* routing that having 2 contracts in the same service host won’t work (for those Wcf fans out there you know what i am talking about). They have to work on a different Wcf hosts.

While helping in migrating an existing Wcf system last week I encountered some of the above. Enter Wcf multi point listener which allow you to do the following:

1- Map multiple Wcf endpoints/contracts/hosts to a single replica.
2- Migrate Wcf Services as-is to Service Fabric based hosting.
3- Migrate Wcf Services (and use Service Fabric State) in them.
4- Control over how listening address are created.
5- control over how bindings are assigned.
6- Control over Wcf Hosts, Wcf Endpoints as they are being created. to add behaviors, dispatchers etc..
7- Support for ICommunicationXXXX Service Fabric interfaces which recycles Wcf channels between callers. Maximum of 1 channel type per host/per endpoint is created at any single time.
8- The service fabric client implementation implements disposable pattern and will abort all open Wcf channels upon Dispose or GC.

The code is open source and published on Git Hub here:

Feel free to clone, use, change.

till next time



On Italian Restaurants, Mafia Bosses & Web Sockets (Part 2)

This is the second and final part of 2 parts article, Part 1 is here

The first part focused on the need to create sequential multi queue Task Scheduler to solve the producer/consumer needs of downstream ordered messages on protocols such as Web Sockets.

This part is not going to be long 🙂

  1. Code for the Web Socket  Server (on top the scheduler) along with the unit tests is published here
  2. In brief: The web socket server is designed to work as a stand alone server with Owin (including self hosting) and to work on top of Service Fabric.
  3. The web server uses “Session Manager” to allow the hosting process to interact with active connected web sockets (such as find a socket, send a message down stream).
  4. The web socket themselves are strong types, for example a Customer socket type, an Order socket type and so on.
  5. The web socket supports, close (async), Abort, and Drain and close (async).

please go through the for more details and usage samples.


till next time


Azure Service Fabric: Learning Materials & Literature

Bookmark this page, I will keep it updated with hand-picked Service Fabric learning materials & literature

Added on August 4th 2015