Abstraction Layered Architecture

There is a meme in the software industry that says that the complexity of software must be some function of its size. This need not be so. With proper use of abstraction it is possible to have complexity that is constant regardless of program size. ALA makes use of this.

This is a qualitative graph comparing the complexity of an ALA application with that of a big ball of mud and an average loosely coupled application. This is further explained in chapter seven here.

The maintainability effort over time should qualitatively follow the green curve in the graph below because as software artefacts are written, their reuse should reduce the effort required for other user stories. Product owners seem to have an innate sense that we manage to organise our code such that this happens. That is why they get so frustrated when things seem to take longer and longer over time, and they often ask us "haven’t we done this before". In practice, too often we follow the red curve. Maintenance eventually gets so difficult that we want to throw it away and start again. We reason we can do better. My experience is that we don’t do better when we rewrite. We just create another mess. It is just a psychological bias on the part of the developer caused by a combination of a) the Dunning Kruger effect and b) the fact that it is easier to read our own recently written code than someone else’s.

If we apply all the well known styles and principles, the best we seem to be typically manage is the orange curve, which comes from the COCOMO models, and which still has maintenance effort continuously increasing.

When we did an experimental re-write of a legacy application using ALA, and measured its maintainability attribute, it comes out as improving over time by several different measures.

ALA is based on the theoretical architectural constraints needed to follow the green curve.

In ALA all code is testable. ALA makes it clear when to mock and when to test with dependencies in place. All dependencies are left in place, because all dependencies are design-time or knowledge dependencies.

Therefore, when testing the application layer abstractions, they are tested with their domain abstraction dependencies. In other words, testing the application is acceptance testing.

Testing domain abstractions is easy with units tests because abstractions are zero-coupled. Mocks objects are wired to ports.

From early on in my career, I experienced the CRAP cycle, not so much rewriting applications, but trying to avoid the mess when writing new ones. When starting from a blank piece of paper, I would research all the architectural styles and principles. I would come across things like 'loose coupling', and I remember asking myself, yes but how does one accomplish that? Each time I would still fail.

I started searching for a pre-worked, generally applicable, 'template architecture' that would tell me what the organisation of the code should look like for any program. I searched for such a thing many times over a long career and never found one. Some would say that this is because the highest level structure depends on project specific requirements.

Finally, near the end of my career, I have that template meta-structure that’s applicable to all programs. The turning point was when I noticed two (accidental) successes in parts of two projects. These successes were only noticed years later, 15 years in one case and 5 years in the other. They had each undergone considerable maintenance during that time. But their simplicity had never degraded and their maintenance had always been straightforward. It was like being at a rubbish dump and noticing two pieces of metal that had never rusted. "That’s weird", you think to yourself. "What is going on here?"

One of them had the same functionality as another piece of software that I had written years earlier. That software was the worst I had ever written. It was truly a big ball of mud, and maintenance had become completely impossible, causing the whole product to be abandoned. So it wasn’t what the software did that made the difference between good and bad. It was how it was done.

Analysing the common properties of those two code bases, gave clues that eventually resulted in a theoretical understanding of how to deal with complex systems. This meta-structure is what I now call Abstraction Layered Architecture.

Subsequently, I ran some experiments to see if the maintainability and non-complexity could be predictably reproduced. These experiments, which have worked spectacularly well so far, are discussed as a project at the end of each chapter.

At first this code wont look that bad, but that’s only because the whole program is so small. It looks modular, but you still have to read all of it to understand any part of it. That’s possible for small programs, but of course that strategy won’t scale up.

As we are taught to do, different responsibilities of the thermometer implementation have been separated out into smaller pieces with smaller responsibilities, although ProcessTemperatures appears to have three responsibilities. The problem is that all the pieces are in some way collaborating to make a thermometer. They are all coupled in some way, both explicitly or implicitly. That’s why we have to read all the code to understand the thermometer. Scale this up to 5000 lines of code, and we will have a big mess.

We are going to refactor the program using the ALA strategy:

The code now begins to be arranged into two abstraction layers, the application layer and the domain abstractions layer. The application is now the only function that knows about being a thermometer. (It is still doing some logic work - the 'for loop' and 'if statement', which we will address soon.)

All the other functions are now more abstract - they know nothing about thermometers - GetAdcReadings, OffsetAndScale, SampleEvery, Filter, FloatToString, and Display. Notice that the word 'thermometer' has been removed from their names, and none of them contain constants or any other references that are to do with a thermometer or temperature.

These abstract functions give you six things:

Now let’s go one more step and create an abstraction to do what that for loop does: This may seem like a retrograde step, but we need to understand this mechanism to move to our final goal of expressing the requirements through pure composition of abstractions. We want to move the 'for loop' out into its own abstraction, but we don’t want to move the code that’s inside it. We accomplish this by putting the code inside it into another function and passing that function to the for loop function:

"func1" is not an abstraction - you cannot give it a name and learn a simple concept of what it does. That’s why I gave it a non-descript name. The content of func1 is cohesively just part of the thermometer application. The name func1 only serves as a symbolic connection within cohesive code - nothing more than a wiring between two points in the program. In this case func1 is immediately below where it is used in the same small file. But as a program grows, these symbolic wirings are always hard to follow. You would need to resort to text searches to find these connections. These types of connections can be numerous and unstructured in larger programs, and the best way to deal with them is diagrams. A line on a diagram is like a symbolic connection between two points, but it’s anonymous and easy to follow. However, this particular one can be dealt with in text form. So let’s go ahead and remove it by using an anonymous function directly as the second parameter of foreach:

It uses the lambda syntax '()⇒{}', which if you are not already familiar with, is worth getting used to. It’s a function without a name, so think of the ⇒ as being instead of the name of the function, the round brackets as the parameters, and the curly braces as the body of the function.

The next thing we want to do is get rid of the while loop, get rid of the indenting, and stop handling the data that is being passed from one function to another. None of them have anything to do with a thermometer. All those intermediate holding variables: adcs, temperatureCelcius, etc are all just symbolic connections. They are too much work when we just want to compose our thermometer from abstractions.

The while loop and all the indenting are there only because we have 'execution flow' tied in with our composition of abstractions. Basically we want to make control of execution flow another abstraction so that the thermometer can be built by just composing abstractions rather than writing executing code.

To do this we will first show how its done using monads. If you don’t know about monads just skip this section as we don’t need this step to understand our final goal. But for those who do understand monads, it is interesting to visit this step to see why the functional programming guys invented them. Then in the following step we will go to ordinary classes with ports instead of monads.

Monads have allowed us to separate execution flow from composition flow. The composition flow is now a pure dataflow paradigm. Data will flow from the ADC to the display, so that is directly represented by the composition. How it executes is separated out, and we will go into how that works shortly. Let’s first understand the 'composition' and why this is so important.

Even if you don’t understand how the monads work, you can see that syntactically the program is now very nice because all it does is compose instances of abstractions, and configure them with constants to be a thermometer. The composition is not declarative - it is dataflow, because dataflow suits how to describe the thermometer. If we let go of how it executes and just trust that the dataflow from one instance of an abstraction to the next works, the program becomes highly readable.

We are using the word 'composition' here to mean the things we are joining together in adjacent lines of code. It can also mean joining boxes with lines in a diagram. Think of a composition as analogous to the adjacent notes in a music score, which are always played successively. If the lines of code are statements or function calls, we are composing things for imperative execution by the CPU. If the lines of code are data processors, we are composing things for successive processing of data. The output of one passes directly to the input of the next.

If we are stuck with thinking in terms of imperative execution flow (the only way of thinking in the C language) we will need to try hard to let that go, and realize that in ALA, 'composition' can be any programming paradigm you want.

Also notice that the first statement just builds the program. Then the second statement sets it running. This two stage aspect of monads is common in the programming paradigms we will use in ALA. It is because the underlying execution flow is not the same as the flow of the programming paradigm. We first wire it up, and then we tell the wired up structure to 'execute'.

There is a second important difference from the while loop version. The while loop version handled the data itself. Each function returned the data which was stored in a local, otherwise useless, variable and then passed into the next function. The monad code doesn’t do that. Instead, it creates and wires together objects which will, at run-time, send the data directly from one to another via an interface. This does not mean that the abstractions themselves know anything about each other - they are still zero coupled. But the application now doesn’t have to deal with data. It just has to compose abstractions.

Lastly, here’s how monads actually execute - the execution model. Don’t worry if this doesn’t make sense.

Each function in the program statement (the function after each dot) executes once at the start. They are not executed when the program is running. Each of these functions first instantiates an object (using new), and secondly wires that object to the previous object.

The functions wire the objects together using an abstract interface. Common interfaces used for monads are IEnumerable or IObservable. These interfaces support iteration of data, by returning an IEnmerator or IObserver. If using the IEnumerator interface, there is a simple method in the interface that pulls data from the previous object. If using the IObseravble interface, there is a simple method in the interface that pushes data to the next object. So IEnumerable/IEnumerator and IObservable/IObserver as abstractions are pretty much just the concept of dataflow, the same abstract concept we will use in the ALA version.

Here is the same program as above, but we are using plain classes with ports instead of monads. We use the 'new' keyword explicitly to create the instances of abstractions, and explicitly wire them together using a wiring function. It’s a little less succinct than the monad version, but the idea of "objects with ports that you wire together like electronic components" is easier to understand, and more versatile. It is necessary for developers to be able to write new domain abstractions, so this needs to be easy.

The wireIn method is doing dependency injection.

The WireIn method returns the new object, so it is possible to string WireIns together. This is called fluent syntax.

Monads are generally not versatile enough to handle multiple ports of different programming paradigms, which we will want in ALA programs. Monads usually only support dataflow. But what if we want to also compose the UI, or event-driven? What if we want to compose transitions between states of a state machine? In ALA, we are able to do all this in the one application, in the same way - using whatever programming paradigms are the best way to express the requirements.

Some instances of abstractions will need to take part in multiple paradigms, such as both UI and dataflow. When we boil down the description of our requirements to pure composition, our composition will often be a graph of relationships. And when you have a graph, your composition is best described by a diagram.

To illustrate this let’s add some UI to our thermometer:

The diagram has both UI composition and dataflow composition. For the UI part of the composition, the lines obviously don’t mean dataflow - they mean 'display inside'. So now different lines in our diagram have different meanings. Here is how that diagram is represented as text.

The text of the ALA thermometer has a symbolic connection for one of the wirings, "temperature". This is ok in this small program, but doing that won’t scale up. That is why we consider the diagram to be the source, and this text version is generated from it.

Looking once again at the diagram, you can see that ALA has allowed us to keep all cohesive knowledge about a thermometer together, and quite succinctly. It contains all the details needed to describe a thermometer, but does so in terms of domain abstractions that are not specific to a thermometer at all. There is no implementation in the application code. All implementation is done by domain abstractions. If you can see that point in the example code, then you are pretty much understanding ALA.

Once we have this diagram, it is easy to conceive how we might add features. For example, we could add two radio buttons into the UI, and wire then to a switcher abstraction that switches the data path between two instances of OffsetAndScale to change between Celcius and Faranheit.

The only unit of code in ALA is an abstraction. Not module, not class, not function. An abstraction may be implemented as a class or interface, (or multiple of them). But an abstraction is more than a module, class or function in that it must represent a 'generalized conceptual idea'. It’s what our brains use to understand the world, not what compilers use. It must be learnable as a concept. Many modules, classes or functions in traditional code are not good abstractions, especially if they are an integral part of a specific system with associations with peer modules, classes or functions.

Abstraction goes hand in hand with reuse. Krueger said that abstraction and reuse are two sides of the same coin. Abstractions tend to be stable - as stable as their conceptual idea. When we have a dependency on an abstraction, it is a dependency only on that conceptual idea.

Here is an example domain abstraction. It works on a stream of data. It smooths the data and slows down the rate. It has an input and an output dataflow port:

In ALA, abstractions are implemented as files. The name of the file is the name of the abstraction. The file typically contains a class, but may contain more than one class, enums, delegates, interfaces etc. I usually put the implementation of the public configuration interface first, then the ports that are implemented as fields, then any other internal state, then the implementations of the ports.

Internally, abstractions are cohesive (which means sticking together) or highly interconnected code collaborating for a single purpose. Because it’s cohesive code, ALA does not prescribe any internal organisation for abstractions. We do not care about dependencies or any inter-relationships inside an abstraction. Any part of the code may need knowledge of any other part to understand. In fact the more cohesive everything is the better, as long as it is small.

Abstractions are the only mechanism that provides design-time information hiding. When David Parnas coined the term information hiding he meant at design-time. Unfortunately there is a popular meme that information hiding means encapsulation, which only hides at compile-time, not necessarily at design-time. The same idea of design-time information hiding has other names such as Alistair Cockburn’s protected variations, and Robert Martin’s version of the OCP (open closed principle).

In the case of our LowPassFilter, the quality of being a good abstraction means that its internals are not coupled or collaborating with other modules. It’s bounded by its conceptual idea. If it were a module instead, it might primarily perform the filter function for a specific application, but knowing which application it is part of may cause it to take on ancilliary functions, such as offset and scale to output the units needed by the application. Or it might grab its input knowing where it comes from. Or it might send it’s output to a specific place, or even to multiple specific places using direct method calls. It might connect to the application’s settings menu for its settings. All these things would cause design-time coupling through the application.

Internally an abstraction is cohesive. This means that every line of code is related to, and collaborates with, every other line of code to implement the idea of the abstraction. It has no internal structure imposed by ALA. It is essentially a small ball of mud.

We shouldn’t think of an abstraction implementation as being a composition of elements such as functions, variables, structs, methods, enums, delegates, statements, lines of code or the like because they are all collaborating. They are all just syntactical components, but not semantic components. In other words, they don’t have meaning on their own related to the concept of the abstraction. Only when we take all of lines of code together do we have meaning. So we never semantically break an abstraction implementation into smaller parts.

The external structure and internal structure with respect to abstractions are at opposite extremes. One is zero coupled and constrained only to the use of instances of lower layer abstractions and the other is fully coupled and highly cohesive.

That there is no structure in an abstraction’s implementation is important because in conventional architectures we are used to hierarchical encapsulation. In ALA there is no containment nesting at all. Instead there is only abstraction layers.

This is the second of the three fundamental constraints. ALA uses a single type of relationship - a dependency on an abstraction that is more abstract than the one whose implementation uses it.

In terms of our previous example, LowPassFilter is Abstraction A and IDataFlow is Abstraction concept B.

Notice how in the diagram the relationship arrow comes from inside A. This is significant. It is the code that implements A that requires knowledge of abstraction concept B. We never actually draw lines when using abstractions, so you wont see this type of line in ALA diagrams. We only use it here while describing this one type of relationship that we are allowed.

B must be more abstract than A. "More abstract" means more general, not specific to A, and have a greater scope of reuse than A.

Because B is more abstract, it is more stable. ALA therefore automatically enforces the Stable Dependencies Principle.

The relationship means that, to read and understand the code inside A, you must know the abstraction concept B - not how the insides of abstraction B work. The word "abstraction" implies that it should be learnable in a short time and easy to retain. When we learn an abstraction concept, there is a moment when we suddenly 'get it', a small moment of insight.

In our A & B example above, the code inside B knows nothing of A. The code inside A, while it knows about the concept of the abstraction B, knows nothing about the code that is inside B. So there is zero coupling between the code that implements the two abstractions. ALA is basically a constraint to force us to always use zero coupling.

Here are some more legal dependencies, this time in text form. They are from a user story in the application layer to the domain abstractions layer.

Legal dependencies from a Thermometer user story abstraction on abstractions AnalogInput, LowPassFilter, OffsetAndScale, Display and WireIn. The user story takes input from an analog to digital converter, filters them to remove noise, scales them to be in degrees, and displays them.

Because of the way our brains have evolved to understand a complex world in terms of abstractions, abstractions are the only mechanism that hide knowledge at design-time. Dependencies on more abstract abstractions have zero coupling between the code using the abstraction and the code implementing the abstraction.

Dependencies on more abstract abstractions also explicitly tell us what knowledge we need in order to understand code. For example, if abstraction A is thermometer and abstraction B is display, then to understand the code that implements the thermometer requires knowledge of the concept of display. That’s why we sometimes refer to it as a knowledge dependency. Such a dependency always applies at design-time.

The dependency usually applies at compile-time and run-time as well, but not necessarily. It could be purely a knowledge dependency, for example a dependency may be on a convention, or it may be simply the knowledge of ALA. You need knowledge of those things to fully understand the code. We want to be explicit and clear about knowledge dependencies for every bit of code inside every abstraction, so we endeavour to always state implicit knowledge dependencies in comments. For example, in the implementation of an abstraction for a hardware device, I always include a reference to the datasheet.

Architectures generally work by applying constraints that restrict the infinite variety of ways code could be written. The ALA constraint of allowing a single relationship type may seem severe, but after some practice, it changes from a hinderence to being an increasingly helpful guide to the design. Sometimes the abstractions come easily from the wording in the requirements and sometimes they require a determined inventive thought process, especially in a new domain.

If you are finding it difficult, then first make sure you have well written requirements that capture all details. The user story abstractions should be just a more formal expression of them in terms of abstractions they probably already mention.

In most domains, I usually start with the UI description in the requirements as they most readily reveal their domain abstractions. Then it becomes apparent that dataflows must be connected to these instances of UI abstraction elements. Data transformation and state abstractions will become apparent next.

In describing example layers, we start with some code. This code will build into a complete running application so no holes are left in understanding how everything works. The accompanying bullet points then explain the organisation of the code in the layers. The code is available here: https://github.com/johnspray74/Thermometer

To understand ALA code, you need to have knowledge of the abstraction concepts it uses. In this top layer code, these are:

Once you have knowledge of these abstractions, notice that the application code is readabable by itself. That knowledge sits at the abstraction level of the requirements. It is highly cohesive - every line works together with every other line to make a thermometer. This abstraction holds all knowledge about thermometers. There is no code anywhere else that has anything specific to do with thermometers. It does none of the work itself - it just assembles and configures instances of the the needed domain abstractions.

The application layer is three things in one: The architecture design, the expression of requirements, and the executable. In conventional software development, these are three separate artefacts.

Execution typically occurs in two phases (similar to some monads). In the first phase the application wires together instances of abstractions. In the second phase the network of instances executes (which is what the finalizing call to Run starts).

At the beginning of this chapter we had an example of a domain abstraction, LowPassFilter. Here is another example:

Notes on the code:

For completeness, here are the other two domain abstractions that we used in the Thermometer application example:

Note that the output port of ADCSimulator is IDataFlow<int>, and the application wired it directly to a LowPassFilter whose input port was type IDataFlow<double>. So we should have wired a type converter abstraction inbetween. We’ll do that in a later example.

As can be seen in the above examples, the domain abstractions layer contains abstractions that can be composed into applications. These are typically building blocks for I/O, data transformations, and persistent state, but many other types of abstractions are possible.

It’s important that these domain abstraction no nothing about the thermometer. They are not specifically parts of a thermometer. Even though in our example, an instance of an ADC is composed with an instance of a LowPassFilter, the ADC and the LowPassFilter don’t know who they are going to be wired to. They could be wired to anything that will give them data or take their data. They only know as far as their own ports, not beyond that.

Some of these types of abstractions may also be found in general language libraries but domain abstractions are more specific to the types of applications we want to express using them. They are specific to a domain, making them more expressive, but less reusable than general purpose library abstractions. They are still reusable both within a single application and by other applications in the same domain.

Another difference between ALA domain abstractions and typical library abstractions is the use of ports. This supports building functionality by simply composing instances of abstractions. While the higher layer composing code controls who will communicate with whom, it does not actually handle the data at run-time. The data moves directly between instances via the wired ports. Library abstractions typically don’t use ports. So the composing code typically has to handle the data. They will pass data to the library abstraction instance, and receive data back.

A further difference between ALA domain abstractions and typical library abstractions is that when domain abstractions are composed, the meaning of composition can be a programming paradigm other than imperative. For example the application above uses a dataflow programming paradigm. Imperative is not usually a good programming paradigm for the expression of requirements, but it’s all your basic language and library gives you (unless you are using language integrated monads, which also give you dataflow programming). But ports allow you to use any kind of programming paradigm. For example, we can make ports that when wired mean laying out a user interface. That is, wiring these types of ports means putting a UI element inside another UI element on the display.

The domain abstractions layer and programming paradigms layer together are like a DSL (Domain Specific Language). We can think of domain abstractions as composable elements and the programming paradigms as the grammar.

The domain and programming paradigm layers are an internal DSL (as appossed to an external DSL). This means that they only use the underlying language (WireTo is an extension method, and new is a language keyword). They don’t introduce any new syntax outside of the underlying language that would need parsing. This allows you to easily fall back on the greater flexibility of the underlying language when you need to. For example, when configuring a domain abstraction, you could pass in lambda expressions. Or, you can pass a whole object of a class that you write in the application layer (which is the strategy pattern).

Conventional libraries generally contain good abstractions. What makes them good abstractions is that their designers don’t know anything about the specific applications that will use them. Writing domain abstractions is best done in the same way. After the need for an abstraction is decided, pretend you don’t know anything about the application, and are writing something to be useful, reusable and learnable as a new concept. Then they should not just be specific parts of a specific application, but concepts that you can learn so that when you read that code that uses them, you don’t have to go and read the implementation.

As we said, abstractions know nothing of each other at design-time, yet can still communicate directly with one another at run-time. It is the responsibility of the code in the layer above that composes them to know the meaning of the data. It knows the meaning of the data even though it does not handle the data. For example, our application above knows that data going from the ADC to the LowPassFilter is raw adc values representing temperature, and what the values mean. It knows that the data passing between the LowPassFilter and the OffsetAndScale is sufficiently smoothed and slowed down for a stable display. It knows that the data passing from the OffsetAndScale to the display is in degrees Celsius. All these pieces of knowledge are cohesive in the design of the thermometer, and are contained together inside the Thermometer abstraction. In a conventional application, these pieces of knowledge would likely be spread between multiple cooperating modules.

For the Thermometer example application above, we wired four instances of domain abstractions. That wiring used a single programming paradigm, Dataflow. Here is the interface for the dataflow programming paradigm:

Programming paradigm interfaces are often this simple. Another example programming paradigm is synchronous event driven. The corresponding interface might be:

Here is the interface for another common programming paradigm, the UI layout. In this programming paradigm, a parent UI element is wired to its contained child elements.

The interface returns the .NET WPF element of the child. This allows domain abstractions to construct the UI using underlying WPF elements. In effect, UI domain abstractions are wired together in a similar way to XAML. Both have descriptive, tree structured syntax, but ALA is using the underlying C# language whereas XAML is using XML.

You can conceive other programming paradigms to give different meanings to composition of domain abstractions in such a way as to allow the easiest representation of typical requirements. For example, for implementing game scoring, I used a programming paradigm called ConsistsOf. For tennis, I used it to express that a Match consists of Sets, a Set consists of Games, and a Game consists of Points. See the example project at the end of chapter four which uses it to implement both tennis and bowling scoring programs. Being able to compose instances of domain abstractions together with meanings that you define in the programming paradigms layer is a powerful way to express requirements succinctly.

ALA is polyglot in programming paradigms. An application typically mixes a small set of different paradigms. Each provides a different meaning when the application wires two instances of domain abstractions together. Examples might be Dataflow, UI Layout, Event driven, State machine transition, Data schema entity relationship. Some may have variations such as pushing or pulling, or synchronous and asynchronous.

Programming paradigms provide the execution mechanism of direct communication between instances of domain abstractions. They do this without the abstractions themselves knowing anything about each other. This means that it is the programming paradigms that control the whole way the application actually executes. Neither the Application layer, nor the domain abstractions layer are involved in this. The application layer is just composing domain abstractions. The domain abstractions only respond to what happens at their ports.

Execution models such as synchronous vs asynchronous, push vs pull, and fan-out and fan-in wiring are discussed in Chapter four.

Programming paradigms provide the rules for the way instances of domain abstractions can be composed by the application. They are the grammar of the DSL.

Through the use of programming paradigms, domain abstractions know nothing about each other at design-time, yet instances of them can communicate at run-time. Of course we could achieve this by making the application handle the run-time communications. The common example would be an application that calls one function which returns a result, and then calls another function passing the result to it. This would almost comply with ALA. The problem is we don’t really want the application to be concerned with run-time communications. The application really just wants to concentrate on representing user stories by composing instances of domain abstractions. So we use a layer below the domain abstractions called programming paradigms. The programming paradigms allow domain abstractions to have compatible ports, which in turn allows them to communicate directly with one another at run-time. The design of the specific communications that will take place is in the Application layer, but all the execution is in the domain abstractions and programming paradigms layers.

The foundation layer contains code used to support ALA programs in general.

A common pattern I use is a WireIn() and a WireTo() extension method in the foundation layer. The application layer uses them to wire together instances of Domain Abstractions using ports.

WireIn() and WireTo() are dependency injection methods, but ALA is more constrained than your average application implemented on dependency injection, because the interfaces used must be more abstract than the domain abstractions whose instances are being wired together.

WireTo(), uses reflection. It’s not essential to use reflection for ALA. You could use dependency injection setters in every domain abstraction instead. You would need one setter per port on the left abstraction. You wouldn’t use constructor dependency injection because sometimes wiring a port is optional. I prefer using the WireTo extension method because it allows domain abstractions to not need all these setter methods.

Here is minimal code for the WireTo method in case you are interested to see how the reflection works. It not necessary to understand this code to use ALA.

Four different exceptions may be thrown. 1) Object A (left object being wired) is null. 2) Object B (right object being wired) is null. 3) A specific A side port was specified, but it is already wired. 4) No matching A side port was found.

Once again, slightly more complete code is available here: https://github.com/johnspray74/Thermometer

If a single abstraction is used for the application, then as more and more user stories are added into it, it will eventually get too large for the ALA size constraint. Meanwhile, domain abstractions and programming paradigms are stable and do not generally grow larger with overall program size. They may increase in number, but it is the application that will go over the 500 line complexity limit.

ALA will need to be applied to the large application abstraction by adding a new layer below it. The requirements are likely already written in terms of naturally separate abstractions which we call features or user stories. We can use these abstractions as the basis for the new layer. We call the layer "Features" or "User Stories". The application abstraction becomes a composition of features or user stories.

Let’s change the Thermometer example application we used above to have a new feature for measuring load as well as temperature. At the same time, let’s introduce a features layer with the two features: temperature and loadcell.

I have deliberately retained a need for communication between the two features to show how features can also have ports and be wired together.

If you see an ALA application, you will find three to five folders that correspond with abstraction layers (described in the previous section). For example:

Knowledge dependencies only go down these layers. So to understand the code inside files in the higher layers, you need to have knowledge of what all the files in lower layers do. There are no dependencies between files in any folder.

Namespaces exactly correspond with these folder names. Therefore we have namespaces called "Application", "DomainAbstractions", "ProgrammingParadigms", etc. This tells us which layer an abstraction comes from, and which folder it resides in.

Namespaces also avoid naming conflicts between layers. They are not useful beyond that. Unfortunately, there is no convenient way of telling the compiler or the IDE to not 'see' classes, interfaces etc in other files in the same namespace or folder.

Abstractions are implemented as files. Abstractions are often implemented as a single class, function, or interface, but sometimes an abstraction consists of a small cohesive group of them, with things like delegates, enums, or even variables. Internal to an abstraction, they interconnect with each other unconstrained. There are no dependency rules inside a file. The only constraint ALA makes is that the total must be small - of the order of 200-500 lines of code, or under the brain size limit. This applies to all abstractions, including the ones that make up the application itself in the top layer.

In time I hope languages catch up and give us visibility support for ALA abstraction. This would probably involve a new construct called Abstraction{} to group the elements of an abstraction. It does not need a name. This construct replaces the use of a file that we are currently using as a stand-in. Anything public inside the Abstraction is only visible to code in higher layer abstractions, i.e. abstractions in higher namespaces. It is not visible in your own namespace, nor to those of lower layers. The compiler would need to know the namespace layering order. If we had this, we would have compiler checking for illegal dependencies.

In ALA, abstractions are usually a single class. Instances of such abstractions are objects. It is the objects that get wired togther by their ports. Classes are the design artefacts that know nothing about one another. Objects are the run-time artefacts that communicate with one another at run-time.

Classes have a 'main interface', the constructors, and any public methods and properties. A class can also implement other interfaces. In ALA, a class’s 'main interface' (it’s constructors and public methods and properties) are only used to instantiate and configure the class from a higher layer. It is never used to actually use the class to do its work. This is effectively the ISP (interface segregation principle). The client who instantiates a class object is different from the classes whose objects will interact with it, so different interfaces are used.

Only the higher layer with knowledge of the system has the relevant knowledge of what should be instantiated, how it should be configured, and how the instantiated objects should be composed together to make a system.

The 'main interface' of a class is 'owned' by the class and is specific to the class. This may sound like stating the obvious, since it is there to allow instantiation and configuration of said class. The thing is that no other interface implemented or required by the class can be 'owned' by the class. No other interface can be specific to the class. The class may not provide an interface designed specifically for it, not may it require an interface designed specifically for it. In other words, all other interfaces must be abstract and in a lower layer.

The idea that classes may not own any interface except the 'main interface' is critically important. If a class were to own another interface that is used for the class to do its work, then that interface would inherently have in its design knowledge about the class. This is true whether the interface is a provided interface (for other classes to use) or a required interface (for other classes to implement). Required interfaces are common for example in 'clean architecture'. They are illegal in ALA.

The inherent knowledge about the class contained in 'owned' interfaces will cause coupling. A class using an interface provided by another class will have design knowledge of what that other class provides at the same abstraction level as itself. It will be written according to what is being provided. There will be a fixed arrangement between the two classes. Over time, this fixed arrangement will cause a blurring of their respective responsibilities.

A class implementing an interface that is required by another class will have a similar problem. It will have design knowledge of what that other class requires at the same abstraction level as itself. It will be written according to what is required. There will be a fixed arrangement between the two classes. Over time, this fixed arrangement will cause a blurring of their respective responsibilities.

Therefore, classes in ALA do not have association relationships. Instead they just have fields of the type of these more abstract interfaces or they implement these more abstract interfaces. We call both of these ports.

The abstract interfaces that we put in lower layers are obviously have to be be general. It can be hard to see how this could work, but it does. For example, interfaces often implement a set of four methods for CRUD operations (Create, Read, Update, Delete). The very existence of this acronym suggests an abstract interface.

An abstract interface in a lower layer makes it much easier to have multiple classes implement or require them. Objects of different classes can then be composed together in arbitrary ways, giving us the powerful principle of compositionality. (The meaning of a complex expression is determined by the meanings of the constituent expressions and the rules used to combine them.)

Abstract interfaces suggest general meanings for the ways we compose objects. They end up looking a lot like programming paradigms, which is why we call the layer ProgrammingParadigms.

Any given class will typically implement/accept more than one of these abstract interfaces. These are the called ports. When using dataflow they are I/O ports. We do not think of the objects that get wired to these ports as clients. The word client is best used for classes in a higher layer (that use the main interface). The classes of the objects to which an object is wired are just called peers.

ALA can be applied to functional programming too. Abstractions are then obviously functions, and the same ALA relationship restriction applies - a function may only call a significantly more abstract function. The functions then form layers.

Where one function may have called a peer function in conventional code, now a higher layer function that has the system specific knowledge is needed to call the first function and then call the second function, in effect composing with functions. Parameters and return values are effectively port. If the first function called the second function in the middle rather than at the end, the second function will now need to be passed into it. The function parameter is also a port.

A higher layer function may call a series of lower layer functions, passing data from one to the next. We don’t often need the data in the higher level function. All we are trying to do is compose functions. It clutters up the code something awful when we have to handle data without needing to use that data. So that’s why we prefer to compose with objects with ports.

Monads also allow composition of functions without this cluttering. But they only support a dataflow type of programming paradigm. We want to compose using multiple ports on our abstractions using multiple programming paradigms. Objects with ports are a more straightforward way to think about this. Monads are objects under the covers, and this is part of the reason they are hard to understand (although eventually you get used to it). By composing with objects directly, it is clearer what is going on.

There should be a readme file in the root folder that points to this website (or equivalent documentation) about ALA. In ALA, we are explicit about what knowledge is needed before a given piece of code can be understood (knowledge dependencies). To understand an ALA application, you need a basic understanding of ALA (from this chapter). So that’s why there should be a readme file pointing here.

In this perspective of ALA, we view it as a vehicle for multi-paradigm programming.

Many higher level programming models are based on a single programming paradigm. Examples are the dataflow model, event-driven, actor (message passing) model, or ladder logic.

A given paradigm makes it easy to solve some problems but not others. Having a single programming paradigm makes the programming model pure and simple. But it’s just plain awkward for certain aspects of typical problems.

So ALA takes the approach that not only will we be able to compose using multiple programming paradigms, we do it it with the same wiring operators. This allows them to be easily intermixed in the same user story or feature, or in the same application diagram. Furthermore, creating and implementing a new programming paradigm is straightforward.

In the end, we want to attain a certain level of expressiveness of composition. If we are too expressive we wont have enough versatility to vary our applications in the domain. If we don’t have enough expressiveness, we will have to compose too many low level elements to get anything done.

Some examples of programming paradigms that we use frequently are UI layouts, dataflow, schema relationships, state transitions.

Each programming paradigm usually results in a type of port for the domain abstractions. Instances of two abstractions can then be wired by their compatible ports. The programming paradigm provides the meaning of that composition, and provides the execution model for that meaning to be carried out.

There are other types of programming paradigms that don’t need to use ports as well. For example, say you want a programming paradigm for style. You would create a Style concept abstraction in the programming paradigms folder. Then every UI domain abstraction would get its style properties from this abstraction. Then the application layer configures the style abstraction for a specific application, and all UI instances would take on that style. You would only use this method rather than ports if every ports would be connected to a single instance of something. If things are connected to one instance, that indicates that the instance itself can be an abstraction, and simply be put down a layer for everyone to access.

Some will disagree with the last paragraph as it effectively makes the style object a global. That’s not great even if it is a good abstraction. Indeed if you want to say test a UI domain abstraction with styles, and do these tests in parallel, the global wont work. Or there may be leftover state in the global between different tests. Or we may want to override the style on one UI instance. If we don’t want to use global instance of style, then we go back to ports. We then create an instance of style and wire every instance of every UI domain abstraction to this instance. To make such wiring easier, I have WireMany operator. This extension method will look for a compatible port on every instance of every domain abstraction.

In the trivial examples of composition that we already gave, we either used fluent style text for the wiring code drectly, or we manually translated a diagram into fluent style wiring code. You may wonder, why bother with diagrams? They require a tool.

There is a bad meme in the software industry that diagrams and text are equivalent for representing software, They are far from equivalent. They each have different strengths and weaknesses. Using the wrong one will significantly increase the difficulty of reading it.

The only reason I convert diagrams to fluent style text is because our examples are small and they have a mostly linear or shallow tree topology. I manually translate the diagrams to readable text to show how the diagrams execute.

Linear diagrams and shallow trees can better represented in text. Deep trees become hard to read because it results in too much indenting. For larger diagrams with arbitrary cross connections, using text requires a lot of 'symbolic connections' or labels to represent what would be anonymous lines on the diagram. These symbolic wirings make the code much harder to read as their number increases beyond a few connections. If you find yourself doing "all files searches" when reading code, you finding those connections, painstakingly one at a time. If you use a diagram, you just follow the lines.

An advantage of expressing the network in diagram form is that you don’t have to give names to instances. You can leave them all anonymous if you want to. Sometimes you will give them names anyway as documentation. For example, if you have two grids implied in your requirements, you will want to give them names so you know which is which in the diagram.

Avoiding diagrams is like an electronics engineer avoiding a schematic, or an architect avoioding drawings. In conventional code, a network of relationships implied by the requirements is still in there. If we use text to represent it, it is much harder to read, even if we put it in one cohesive place. But what we actually do is even worse. We distribute that text throughout our modules, making it difficult to see. That is why conventional code typically becomes a mass of dependencies resembling a big ball of mud. Using such tricks as dependency inversion, indirections, or container style dependency injection makes the situation even worse. The network of relationships is still there, but now it is even more difficult to see.

Sometimes programming with diagrams is called model driven software development. I prefer not to use the word 'model'. In the real world, models leave out details. Software models tend to leave out details too. ALA diagrams do not leave out details. All details from the requirements are represented, for example in the form of configuration of the instances. That no details are left out is why the diagram is also the executable.

The diagram is stored in the application layer folder. When a diagram is used for the internals of a feature or user story, it resides in the respective layer folder. When diagrams become large, they need two tools. One tool allows you to draw the diagram, and the other generates wiring code automatically. The generated code does not use fluent style - it is just a list of instantiations followed by a list of wirings between them. The wiring code generated from the diagram lives in a subfolder from where the diagram is, because it is not source code.

If manually generating code from a diagram, the diagram should always be changed first, then the code. There should be a readme explaining exactly what the generated code should look like.

Automatically generated code does not need to be readable except to the extent of finding where it doesn’t accurately reflect the diagram.

Because a decomposed system tends to create modules that are specific to the system, these modules tend to be not reusable. They may be replaceable with modules that have the same interfaces, but not actually reusable. We tend to encapsulate such modules inside the system. Similarly with submodules, we encapsulate them inside their modules. This creates an encapsulation hierarchy. It is sometimes likened to a map in which we can zoom in for greater detail.

This actually doesn’t work for hiding information at design time. Because the modules are specific and not abstractions, you will always have to zoom in for the details of the inner modules in order to understand the system. Encapsulation makes as much sense as hiding the picture on every individual piece of a jigsaw puzzle unless you zoom into it, then trying to see the big picture.

The encapsulation may help to unclutter the IDE namespace at the system level, but it doesn’t reduce how much you have to go inside the encapsulations to understand the system.

If the modules and submodules are abstractions, and those abstractions are more abstract than the modules that use them, then we don’t have to zoom in. We can understand a system in terms of the abstractions it uses.

Abstractions are reusable. So we explicitly do not want to encapsulate them inside something that uses them. We need them to be public for reuse. Instead of encapsulating them, we use abstraction layers.

Decomposition tends to break up a system first according to these types of criteria:

A system decomposed in this way will make features or user stories span the modules. This is bad. It forces us to create dependencies for communications within a feature or user story.

In ALA, features or user stories are obvious abstractions given to us by the requirements. As such we keep them together, even if they cross over these other boundaries. For example, if UI, business logic, and storage span three different machines, there is nothing stopping us coding or drawing a single diagram containing all the elements of the UI, business logic and storage for a user story. The elements will be instances of abstractions deployed on different machines, but that is a deployment detail. As long as the internal lines in the diagram represent asynchronous communications, the feature or user story will still work when deployed. Deployment time abstractions can insert the necessary middleware. No specific interfaces are needed between the elements on different machines because they are instances of abstractions that already have compatible asynchronous-ready ports. (We cover asynchronous ports in detail in chapter four.)

The way the resulting code is organised will be completely different from a convention decomposition. The relationships that exist between conventional modules will disappear. They become cohesive lines of code inside a new abstraction representing the system. In fact that’s all the system abstraction will need to do.

One implementation model for components is so called components and connectors. The mechanics of components and connectors is that the lines drawn between components are connector objects. They contain a value, which is the 'data on the wire'. Thinking of the wire as being a variable with a value is quite a useful programming paradigm. It is also a relatively efficient execution model. The variable itself is a shared variable that isn’t globally visible. Only the two instances of components that are wired together can ever see it. Senders need only set the value of the variable, and receivers need only read the variable.

When two compatible instances of components are composed or wired together using this programming paradigm, the implied connector object is created automatically and wired inbetween them.

The instances of the components must be active objects (somehow execute by themselves).

Let’s see how to create such a programming paradigm for ALA applications:

Let’s create two domain abstractions to demonstrate the use of this programming paradigm. First a domain abstraction with an output port using this programming paradigm.

And another domain abstraction with an input port using this programming paradigm.

And finally an application to wire together instances of these domain abstractions:

Application.cs

Here are two atom abstractions:

Instances can be composed to make a molecule:

If water was implemented in the same way we typically write software, there would be no water molecule per se; the oxygen atom would be modified to instantiate hydrogen atoms and interact with them. Even if dependency injection is used to avoid the instantiating, it is still unlikely that a water abstraction would be invented to do that, and there would still be the problem of the oxygen atom interacting with hydrogen’s specific interface. The oxygen module still ends up with some implicit knowledge of hydrogen. And hydrogen probably ends up with some implicit knowledge of oxygen in providing what it needs.

This implicit knowledge is represented by the following diagram. The relationship is shown coming from the inner parts of the modules to represent implicit knowledge of each other.

While oxygen and hydrogen are modules, they are not abstractions because oxygen is implicitly tied to hydrogen and vice-versa. They can’t be used as building blocks for any other molecules.

To keep oxygen as abstract as it is in the real world, an interface must be conceived that is even more abstract than oxygen or hydrogen. In the molecule world this is called a polar bond. It is one of the programming paradigms of molecules. Its execution model at run-time is the sharing of an electron.

The corresponding software would look like this:

The water molecule has a "uses instances of" relationship with the two atoms, and the atoms have a "uses instance of" relationship with the even more abstract polar bond abstraction. Polar bond is an example of an 'abstract interface'.

The second real world analogy is Lego. Shown in the image below is the same three layers we had above for molecules, atoms and bonds.

The domain abstractions are the various Lego pieces, instances of which can be assembled together to make specific creations. Lego pieces themselves have instances of an abstract interface, which is the stud and tube. This is a programming paradigm. There is a second abstract interface, the axle and hole. These programming paradigms have an 'execution model' at run-time. The execution of the stud and tube programming paradigm is to hold structural integrity. The execution of the axle and hole programming paradigm is rotation.

The third real world analogy comes from electronics. The abstractions are electronic parts, instances of which can be composed as a schematic diagram:

In this domain, we have at least two abstract interfaces as programming paradigms, one for digital logic signals and one for analog signals. Their execution model at run-time is continuous-time voltage levels.

Our forth and final real world analogy is a clock. In this diagram, we show the process of composition of abstractions to make a new abstraction. The process is a circle because instances of the new abstraction can themselves be used to make still more specific abstractions. Each time around the circle adds one layer to the abstraction layering.

Let’s go round the circle once. We start with abstract parts such as cog wheels and hands. Instances of these have abstract interfaces as programming paradigms. Their execution models allow them to interact at run-time, such as spinning on axles and meshing teeth. The next step is to instantiate some of these abstractions and configure them. For example, configure the size and number of teeth of the cog wheels. Next comes the composition step, where they are assembled. Finally we have a new abstraction, the clock. Instances of clocks can in turn be used to compose other things such as scheduling things during your day. Because you have now created the abstraction clock you don’t have to think about cog wheels when thinking about how to meet someone at a certain time.

There are many other instances of this pattern in the real world, and in nature. In fact everything appears to be actually composed in this way. At least that’s the way we understand and make sense of the world - in terms of abstractions, which are in composition layers.

Here are the three domain abstractions:

Take a moment to look at these three abstractions:

 — The UI has a lamp you can control, and a push button which outputs an event (should have been two separate abstractions).

 — There is a WarmerPlate. It tells you whether or not a container is on the warmer plate, and whether or not it is empty. It controls its own heater.

 — There is a Boiler. It can be turned on or off. It will tell you when it is empty of water. And you can stop water flow instantly with a steam release valve. It will turn its own heater off if it runs out of water, or the valve is opened.

That’s all there is to know about the three domain abstractions.

We have three programming paradigms

 — live dataflow (works like an electronic circuit)

 — events

 — simple state machine

The API for the Programming Paradigms layer is described in the key on the right of the diagram below. It gives you all the knowledge from this layer to be able to read the diagram. So, for example, a solid line is a dataflow; the rounded box is state with the states enumerated inside it.

The details of how to turn the diagram into code is explained in a project document, also provided in the Programming Paradigms layer.

Now that we have understood the knowledge dependencies in all lower layers, we can read the diagram that resides in the top layer, the application layer:

The diagram to the left is the application itself. Instances of the three domain abstractions, UI, Boiler and Warmer plate are shown as boxes.

Follow me now as we go through the user stories by looking at the lines on the diagram:

That’s all there is to reading this application. The code for the coffee machine can be read and understood in about one minute. Compare that with reading other solutions to the coffee machine problem.

Note that the paragraph above is pretty much a restatement of the requirements in English. It could have been the requirements. The amount of information in the English form (or the diagram form) is about the same, thus the Domain Abstractions gave us the correct level of expressiveness. Further confirmation of this is if the level of expressiveness allows us to modify it.

For example, say a requirement was added that a coin device was to enable the machine to be used. The coin device is an abstraction that provides an output when a coin is given, and has a reset input. Looking at the diagram, and being able to reason about its operation so easily, you can see that the coin device’s output would intercept the Pushbutton using another instance of an AND gate. And to reset the coin device, you could use the boiler empty output event.

To make it actually execute, we apply the manual procedure documented in “Execution models.doc”. This document is in the Programming Paradigms layer. It will generate these 6 lines of code:

There is a one-to-one correspondence between the lines in the diagram and the lines in the code.

As you can see, the execution model is a simple one. The 6 lines of code are continually executed in a loop. This execution model is effective and appropriate for this small application.

The 6 lines of code can be built into a complete program shown below:

If you are using a diagram as we are in this solution, you always change the diagram first when the requirements change. It provides the expressiveness needed to see the application’s requirements represented in a clear, concise and cohesive way. There the logic can be ‘reasoned’ with. It is not documentation, it is the source code representation of the requirements, and executable, both important aspects of ALA.

The next step is to implement the three abstractions. These are straightforward using the same execution model as was used for the application, so are not shown here.

The resulting application passes all of Martin’s original acceptance tests plus a number of additional tests of behaviour gleaned from his original text.

The maintainability quality attribute is often thought of in terms of ripple effects of change. I don’t think that is quite the right way to look at it. I have often had to make changes across a number of modules in poorly written code. The changes themselves just don’t take that long. The problem I see is the time you have to spend understanding enough of the system to know where to make a change, even if it ends up being just one line of code in one place. To make that small change with confidence that it wont break anything can take a long time understanding the collaboration between modules. You may have had to understand a lot of code to figure that out. You have to understand all the code that is potentially coupled to that one line of code, which is essentially the complexity.

Unlike encapsulation which works at compile-time, abstractions hide complexity at design-time. They give boundaries to how far you have to read code to understand it.

If you are going to have abstraction, it makes sense that you would have instances. An instance is nothing more than the use of an abstraction by referring to its name. If your abstraction is a pure function, then an instance is just using the function, or getting a reference to it.

If your abstraction is a class, and if that class contains data, then you need to instantiate the class so that each instance has its own data. Object oriented languages of course already have these two concepts as classes and objects.

Many discussions on software architecture seem to combine them into one term, such as modules or components. These terms may implicitly contain the separate concepts of abstractions and instances, or they may be intended to have only one instance, in which case it can’t be an abstraction. Not having explicit terms, like class and object, will inevitably lead to confusion. In ALA the terms we use are abstractions and instances.

The problem is that when we become vague about the difference between abstractions and instance, we will then create dependencies between abstractions such as to get or put data. If you create dependencies between peer abstractions, they are no longer abstractions. Instead you need to wire the instances. If we don’t have two separate and clear terms for abstractions and instances, we will end up with no abstractions. All architectural styles based on a 'divide and conquer' methodology appear to have this problem.

A confusion comes from the UML class diagram, which already has the separate concepts of classes and objects. However it actually encourages you to create relationships between classes, destroying them as abstractions. The most important potential idea that OOP brought us was the idea of classes as reusable abstractions, and objects as their instances. It never happened in part because of the UML class diagram, and the very harmful habit of putting dependencies between abstractions instead of wiring instances.

An important property of abstractions is that instances of abstractions can be composed or assembled in an infinite variety of ways. We call this composability. We discuss composabilty in detail in a section below.

Even the tiniest amount of coupling between two abstractions completely kills composability, meaning they are no longer abstractions. So one test of abstractions is composability of their instances. If any two potential abstractions must work together, then they are not abstractions, they are just modules.

ALA has mutual zero coupling between the code inside (the code that implements) all abstractions. This is the case both horizontally between peers in the same layer, and vertically up or down the layers.

In software development we are only interested in design-time coupling. This means that to understand one piece of code, how much do we need to know about other pieces of code? It is about knowledge, which is about design-time. This is the coupling that matters.

We will use the word coupling to mean design-time coupling. That’s consistent with Larry Constantine’s statement. His reasoning came from identifying how to reduce complexity, the time taken to understand software, and reducing the incidence of bugs.

Here is a typical definition of Loose coupling from the internet: "Loose coupling refers to minimal dependencies between modules, accomplished through strict, narrow, stable interfaces at the boundaries."

This definition of coupling differs from ALA’s in two respects.

Firstly, we are not minimizing dependencies. We are eliminating bad dependencies and maximizing good dependencies (as discussed later).

Secondly, it’s not just about using interfaces. A lot of design-time coupling is what I call collaboration coupling. Collaboration coupling is when one module does specifically what another module needs. Collaboration between two modules is often mutual. It’s characterised by the modules having a fixed arrangement with each other (for example, an MVC type of arrangement). (To understand what we mean by "a fixed arrangement", see the section below on the opposite of a fixed arrangement, which is composability.)

An interface between two modules that have a fixed arrangement with each other may hide some details, but it doesn’t prevent fundamental collaboration between the two modules. When modules have a fixed arrangement with each other, collaboration will tend to increase during maintenance.

Some definitions of coupling are in terms of the ripple effects of change. But even if a change ends up being made in just one place, that doesn’t mean you didn’t have to understand code in multiple places before you could understand how to make the change. I have many time spent a day understanding code only to end up changing one line. So this is not a good definition.

Wikipedia defines coupling as "the degree of interdependence between software modules". It doesn’t really distinguish between design-time, compile-time or run-time coupling, and the given formula for coupling seems to reflect compile-time. We need to think of coupling as a design-time property. It is about knowledge of internals of a module. Compile-time and run-time dependencies do not matter. Only design-time coupling matters.

Abstractions are the only type of modules that allow us to achieve zero coupling.

Unfortunately there is a meme in the software engineering industry that there must be some coupling between modules. The argument goes that if the system is to do anything it must have some coupling between its parts. We therefore hear of "loose coupling" as being the ideal. Using the definition of coupling given above, this is completely incorrect. It’s confusing run-time coupling with design-time coupling. Only design-time coupling does. If I connect an instance of difference table generator to an instance of a printer, that run-time coupling between the instances doesn’t create any coupling between the concept of a difference engine and the concept of a printer. I can still understand either concept’s implementation in complete isolation, even though it’s possible to connect instances of them together to form a working system.

Because of this bad meme, in conventional code we have developed a habit of using dependencies to implement communications. We are settling for design-time coupling to implement run-time communications between different parts of a system. It is not necessary. Part of the problem is that we are seldom taught the difference between design-time and run-time coupling. So I prefer to use the words connection or wiring used for communications between instances of parts in a system.

For example, in conventional code, if function Switch calls function Light, the code inside Switch is coupled with Light. If the light’s abstraction level is about the same as that of the Switch (which it is), then the abstraction of Switch is destroyed. When you reuse it you have to know the internal code brings in a Light as well. To understand the system (a Switch connected to a light), you have to go inside the Switch:

If instead, a separate abstraction called "System" has code inside it like Light(Switch()), then Switch remains a good abstraction whose internal code is now only concerned with how a switch works. The code inside all three abstractions is now zero coupled. Understanding the system no longer requires looking inside Switch:

A similar argument applies if Switch and Light are classes. In conventional code they will commonly have an association relationship. Even if Light is injected into Switch by a higher entity called System, Switch still knows the specific interface of a light (LightOn(), LightOff()). This interface is not abstract enough to prevent Switch knowing about Light, and Switch knowing about the System. If you instead have a class, System, that has code like new Switch().WireTo(new Light()) using a generic interface then all three abstractions are zero coupled.

ALA never uses coupling for connections or wiring between parts of a system. A larger system typically consists of many connections. These connections are typically cohesive, and belong in one place. In conventional code they tend to be distributed and buried inside the modules. A smell is that you are doing 'all files' searches to unravel them. In ALA, they are brought out into their own place as a cohesive new abstraction describing a specific user story or system.

What would be collaboration between modules in conventional code becomes cohesion inside a new abstraction in ALA. A call from one module to another becomes a single line of code inside an abstractoion in a higher layer. Would be collaboration between a group of modules for a single purpose or user story (like MVC pattern) becomes several cohesive lines of code in the higher layer. Those several lines of code are cohesive.

Cohesion also increases for the modules as they become abstractions. An abstraction is closely aligned with the single responsibility principle. We can think of abstraction as a "single concept principle" instead of "single responsibility principle". Using abstractions increases the cohesion of the code that implements the abstraction.

ALA provides no structure for the internals of an abstraction because the code is cohesive. The internals of an abstraction could be described as a small ball of mud, which is why they should be small. There is no such thing as a sub-abstraction. Instead the code is composed of instances of abstractions from lower layers. So in ALA, layers replace what would be hierarchical encapsulation in conventional code structures.

Zero coupling and higher cohesion blocks ripple effects of change, whether in higher layers or lower ones. A ripple stops at an abstraction concept because of the inherent stability of the concept itself.

What can happen though is that abstractions can be improved to be better quality abstractions. Often you can generalize an abstraction to make it more reusable by adding a configuration. The configuration has a default, so it doesn’t affect existing uses of the abstraction (convention over configuration).

In our experience, the most common type of change that still affects multiple abstractions are changes to code conventions. Conventions in the ways abstractions are commented, and their code laid out are effectively abstractions in themselves that live in a bottom layer. So when they change, it makes sense that all abstractions that depend on them change. These conventions will mature over time. Besides, while these types of changes may require a lot of editing, they don’t require simultaneous understanding of multiple modules, which is where the real problem with coupling lies.

During code creation, run-time dependencies are easily introduced, and never seem too terrible at the time as they get the immediate job done. But when they accumulate to hundreds or even thousands of them, as they do in most typical applications, that’s when the system, as described on the left side of the table, just truns into a monolithic big ball of mud.

When you are comparing the left and right sides of the table above, you may be wondering, where did the free lunch come from? Where did the bad dependencies go? Where are the disadvantages on the right side of the table. Is this some kind of magic? How can, for example, the Avg module talk to the ADC module at run-time with no dependency on it, nor any knowledge about it? How can such a program even work? Haven’t I just moved the bad dependencies somewhere else? No, there are no tricks. The only answer is that we have been taught to do programming in a very bad way, and have become used to it when we could do a lot better. The knowledge that Avg will talk to ADC at run-time is there, but it is normal code contained within a new abstraction.

If you really want to find a disadvantage, then it is the need to conceive the abstractions: Disp, Accu, Conv, Avg and ADC. It only works as well as the quality of those abstractions. Effectively we have traded the need for dependency management, and all the complexity that bad dependencies cause, with the need to create good abstractions. Creating good abstractions is a skill that does take time sometimes.

Just to recap, the only dependencies we have used are good design-time or knowledge dependencies:

A dependency on an abstraction is a dependency on the concept or idea of that abstraction. A concept or idea is generally stable. So good dependencies are also dependencies on the more stable.

Even if the implementation details inside an good abstraction need to change, the abstraction concept itself is stable. The application example above is really just depending on the idea of an ADC or the idea of a Display. If the details inside those ideas' implementations change it doesn’t matter. For example, if the ADC silicon is changed, the ADC abstraction implementation can also change. But the application is still just using an ADC.

ALA therefore naturally conforms with the Stable Dependencies Principle (depend in the direction of stability). The SAP is mostly used in relation to packages, but ALA does not use hierarchical encapsulations. (You can use packages, but it’s just a collection of abstractions that get distributed together, not an abstraction in itself.) Here we are applying the SAP at the level of the abstractions themselves.

One of the guidelines sometimes used for dependencies in conventional code is that a class that has high fan-in should not also have high fan-out (also called afferent and efferent coupling). Another is that modules higher in the dependency structure should have low fan-in and those lower in the hierarchy have low fan-out.

The argument goes that a class with high fan-in should have high stability but one with high fan-out would have low stability (presumably because dependencies are thought to be things that cause changes to propagate).

In ALA, all dependencies are on more abstract, more stable, abstractions. Therefore the conventional fan-in and fan-out recommendations are reversed. In ALA, it is perfectly fine, in fact really good to have both high fan-in and high fan-out. It simply means that the abstractions are useful and are getting reused.

If we are talking about bad dependencies in a conventional modular system that are used for communication between modules in the system, of course ALA says we want zero fan-in and zero fan-out, because such dependencies are illegal anyway.

In chapter four we will also talk about fan-in and fan-out. Note that the fan-in and fan-out discussed in chapter four is different. In this chapter fan-in and fan-out is talking about dependencies. In chapter four we are talking about fan-in and fan-out in run-time communications the wiring. In other words one instance’s output port being wired to many instances input ports.

Of course in ALA, with only knowledge dependencies present in the system, and the dependencies needing to go toward more abstract abstractions, you obviously cannot have circular dependencies. Nor would that even make sense. (Recursion appears to require circular knowledge dependencies but actually doesn’t. We will visit that in the last chapter.)

Since run-time communications is not implemented using dependencies, circular communications in ALA is simply circular wiring. This is perfectly fine. In fact circular wiring is very common. (The potential issues of circular wiring at runtime is a separate issue that exists in both ALA and conventional code, but can be dealt with more easily in ALA’s execution models. This is discussed in chapter four.

In both ALA and conventional code, circular communications can be a natural consequence of the requirements. But in conventional software design, run-time communications between modules are frequently implemented with dependencies. Then we realize these circular dependencies are a problem and so we add a rule that we don’t like circular dependencies. This is an attempt to mitigate the problem by forcing the modules to have a very arbitrary layered structure. That structure does not actually exist in the nature of the peer modules themselves. (Many modules will actually have a similar level of abstraction, for example views, business logic and data.)

The forced arbitrary layering structure becomes its own nuisance. Some communications that would naturally be a push have to be changed to a pull. (Pushing means a function or method call with a parameter, pulling means a function or method call returning a value). Whether we use push or pull should be able to depend on performance or other considerations such as sending data only when data changes, or when we want to receive the latest data, or how often the source changes, or on latency, etc. It should not be driven by an arbitrary layering of modules.

So then what happens is that when we do want to push or pull for performance reasons, but we need to go in the reverse direction of the arbitrary direction that dependencies can go, we end up creating an indirection, such as a callback, virtual function call, or observer pattern (publish-subscribe). This indirection further obscures the already obscure communication flows through the system.

ALA simply eliminates all this nonsense. In ALA, communication flows:

Sometimes layers are used incorrectly as partitions. Because of this mistake, there is a meme that we should only have dependencies on the immediate layer below. For ALA’s abstraction layers this is incorrect.

When we write our programs using only good knowledge dependencies, the knowledge needed to understand a piece of code can be the abstractions in all the layers below.

For example, to understand this application layer code:

You need to know all of these things from lower layers:

All of these knowledge dependencies should be explicit, in other words the application folder should contain a readme file explaining all these knowledge dependencies, and link to information about them.

It’s not necessarily the case that all lower layer knowledge is needed to understand something. The application is itself, for example, is an abstraction. There can be many instances of it being used by different users. These users don’t need to understand all the abstractions in all the layers, only the application abstraction by itself.

That concludes our discussion on why the ALA structure works from the point of view of good and bad dependencies.

Expressing user stories as a composition of domain abstractions, as discussed in the previous section, is all well and good, but it doesn’t work without defining what composion means. That’s where programming paradigm abstractions come in. They are composition abstractions.

For example, many applications have displayed values or outputs that need to be updated 'live'. In conventional code, we often write imperative code to implement this live behaviour. The code repeatedly gets data from its source(s), does some manipulation on that data, and updates the output. We really should have a programming paradigm for it. In ALA you think of it simply as dataflow. When wiring together instances of domain abstractions by dataflow ports, the composition represents data flowing. This programming paradigm is not new, of course, it appears in Unix’s pipes and filters, functional programming’s monads, when binding GUI display elements to a data source, LINQ, Reactive Extensions, Labview, and function blocks to name a few. Dataflows are often used on distributed systems because implementation over literal wires is naturally a dataflow. But the paradigm is just as applicable inside monolithic systems. What ALA does is make it easy and natural to implement dataflow yourself every time it is the best way to express requirements. We should never be writing imperative code to implement dataflow inherent in our requirements. ALA makes it easy to wire a network of dataflows.

The same idea applies to the event-driven programming paradigm. It is common these days for GUI elements such as buttons, menu items, etc to have event-driven output ports. But then we often just wire them to imperative methods with a dependency. In ALA you create input ports as well. For example all popup window abstractions such as file browsers, wizards, settings, navigable pages, etc have input ports. The main window has a close input port. Long running tasks that need to be told when to start have an input port. Then you can use the event-driven programming paradigm for composing instances of these types of domain abstractions.

Another programming paradigm is building the UI. Building the UI by composing abstractions is common using conventional libraries these days. The meaning of composition in this case is "containing one UI element inside another". The composed UI structure is a tree. For example XAML does this using XML. I do not like the use of XML for this. What ALA brings is doing all composition in a consistent way. Composition of the UI is done in the same way as the composition for dataflow, or for event-driven, or any other programming paradigm you care to invent. That way a user story is fully and cohesively expressed inside its own abstraction just by wiring instances of domain abstractions that have various programming paradigm ports.

Requirements are what’s left when you factor out all implementation details. This is another way of thinking about executable requirements. As mentioned in the previous section, ALA requires you to build your entire application factoring out all pieces of computing work into domain abstractions and programming paradigm abstractions. So what does the application that’s left in the top layer look like? Well if everything abstract has been factored out, what remains must be details specific only to this application. Essentially these details equate with the requirements.

The application code becomes a formal re-expression of the requirements. There typically be some explicit information there that was only implicit in the requirements, but they were requirements all the same. For example, it may not have been explicitly stated in the requirements that a number displayed on the UI should not display decimal places that are not significant, or just contain noise. Or it may not be stated that a displayed value should not change too frequently - it should be slow enough for a human to read successive values. Developer’s should know these types of implicit requirements and explicitly implement them without it being stated in the requirements.

So the application will end up with an instance of a rounding abstraction and an instance of filter abstraction wired into its dataflow before the display. The application will specify the rounding, the filter bandwidth, and the re-sampling rate when it instantiates these abstractions.

ALA’s succinct expression of requirements discussed above is obviously a form of DSL (Domain Specific Language). Under the broader definition of a DSL, the domain abstractions and programming paradigms layers are a DSL. But ALA is not just a DSL. ALA is fundamentally about organising all code into small abstractions that are in layers that are increasingly abstract. This constrains the organisation of code much more than simply implementing a DSL.

ALA does not pursue the idea of an external DSL (a new syntax), nor even the syntactic elegance of DSLs. It doesn’t try to move application development away from the developer to a requirements team as some DSLs can do. For example, you don’t get a new language such as XAML to express UI structure. In fact, expressing the UI structure in ALA moves away from XML back to code. If moving away from code, ALA uses diagrams because they are more flexible and much more readable than code and even more so than XML.

Seen as a DSL, in ALA you wire together plain old objects or functions while conforming to a grammar. The grammar comes from the 3rd layer programming paradigms and from which classes use which programming paradigm for ports. This grammar defines the rules for their composition.

Diagrams and text are sometimes thought of as equivalent, as if they have a duality like waves and particles in physics. It is said to be a matter of personal preference which you use, and since graphical tools are hard to produce, why not use text? I do not agree with this. From the point of view of how our brain’s work best, they are very different, and each is powerful at its own job.

Consider an electronics engineer who uses a schematic diagram. Ask him to design a circuit using text and he will just laugh at you. Electronics naturally has a network or arbitrary graph structure that is best viewed and reasoned about in diagram form. If you turn a diagram into a textual list of nodes and connections, the brain can no longer work with it directly. It is constantly interrupted to search for symbolic references when it should be free to just reason about the design.

Try designing or reasoning about a non-trivial state machine without using a diagram. Most software systems naturally have an arbitrary network inherent within it.

Text can readily be used to compose elements in a linear sequence. It is excellent for telling stories, because stories are a sequence. Our brains are evolved to understand, and even recall stories as long as each sentence relates meaningfully to the previous and the next.

In sequential code, white space is the normal connector between the elements. Sometimes periods or other symbols are used instead.

Text can also handle shallow tree structures, simply by using indenting. Compilers may use the indenting, or they may use brackets, usually () or {} instead. Interestingly, the brackets work for the compiler, but not for the brain. The brain doesn’t see them without specicly concentrating on them. At a glance, it just sees the indenting. So I personally don’t agree that Python’s significant indenting is a mistake as many do. I think using brackets for the compiler and indenting for our brain in the same code can lead to discrepancies. Fortunately moderne editor tend to keep the two methods in sync.

Ordinary structured programming (if statements and the like) and XAML are examples of tree structured code represented in text. When the tree gets deep, the indenting is too deep for our brains to follow. So text is only suitable for shallow trees.

Text becomes troublesome when there are arbitrary connections across the structure forming a graph. It must be done with matching labels. Most imperative programs are actually not a tree structure because of the connections to variables. They must be connected to the code with labels. Local variables in a small scope are not a problem (a small scope being what will fit on your screen and can be understood all at once by the human brain). It only requires an editor that highlights them. For large scopes we end up spending too much time finding and trying to remember the connections, resorting to many all-files searches. It is a cumbersome way to try to reason about what is usually a reasonably simple structure when viewed as a diagram. (When I use the term 'labels', I am talking about labels that are used for connecting two or more points in the code. These labels are not abstractions. References to the names of abstractions are absolutely fine, and we don’t draw lines for them even if we are using a diagram. We always just use abstractions by their name.)

When we need to compose instances of abstractions in an arbitrary network structure, our brains work much better using a diagram. Our brains not only evolved to be good at understanding and recalling stories, but also spacial graphs, presumably to allow navigation along pathways.

The brain can readily see and follow lines between the instances of the abstractions. Unlike text labels, the lines are anonymous, as they should be. When label connections are used, the labels themselves need an encapsulation scope. Lines don’t need encapsulation. They connect two points with clearly no other code having access to them.

Generally lines connect only two points or ports, but sometimes may connect three or more. To understand all places connected by a label connection requires an all files search. To understand all places connected by lines, the brain just follows the lines, generally a short distance on the diagram. The spacial positioning of elements in a diagram is also something the brain readily remembers better than where things are in text. So, diagrams can qualitatively do things that text simply cannot.

If a lines connects together many ports, it is a smell that a new abstraction may be waiting to be discovered. For example, in a schematic diagram, if there are a high number of lines all connected together at zero volts, that’s the ground abstraction. Electronic engineers will use a ground symbol to represent that abstraction instead of drawing long line everywhere. An example in software might be the game score if most instances of domain abstractions interact with it. In this case 'GameScore' could be a domain abstraction in the layer below instead of making it a domain abstraction and wiring every other domain abstraction to it.

The three ALA architectural constraints do not require a diagram per-se. It only requires abstraction layering, and it’s quite possible for a user story to just consist of a linear sequence of instances of abstractions. For example, a sequence of movements by a robot or a "Pipes and Filters" sequence of operations on data. However, ALA is a polyglot programming paradigm because user stories will generally inherently contain multiple programming paradigms: UI, event-flows, dataflows, state machines, data schemas, etc. These aspects of a user story tend to be naturally interrelated, which is what causes the resulting relationships among its instances of abstractions to be a graph. The use of diagrams embraces the idea of bringing together of all these different interrelationships of a user story in one cohesive place.

If dependency injection is used to implement the wiring, I prefer not to use XML to specify the application. Firstly XML is not very readable. Secondly it only handles shallow tree structures well, not arbitrary graphs. If I use text for specifying wiring, I use normal code. I try to find a tree structure in the graph as much as possible and represent that as indented text as much as possible. Any nodes in the graph that need cross-tree connections have their instances saved in local variables. The cross connections can then be wired by referring to those variables. You will see this done in many of the examples.

You are still better off with this code in one place than having it distributed inside your modules. But if a graph structure is inherent in the requirements, there is really no substitute for the readability of a diagram.

The ALA design process is describing your requirements using abstractons and inventing the needed abstractions as you go. It is an intense intellectual activity, especially when doing it for the first time in a new domain. As well as expressing your user stories, you are inventing a set of domain abstractions and programming paradigms that will allow you to express all user stories. It requires all your focus. There is no focus left to deal with tools that are hard to use.

I have found that hand drawing the diagram on paper is not good. The diagram quickly gets into a state where it needs reorganising, which requires redrawing it from scratch. That totally interrupts your design flow. I have found that a diagramming tool that constantly needs you to control the layout, such as Visio, is also not good. The tool must be able to reorganize the diagram (push things out to make room).

So until there is a better tool, I have been using Xmind because as a mind-mapping tool, it is designed to not get in your way as you are creating. It lays itself out as a tree structure, and then allows cross connections on the tree to be added using a key short-cut at the source and a mouse click at the destination node. It has serious limitations, however I use some simple conventions to mitigate these. For example, I use '<' and '>' to represent input and output ports.

Furthermore, the tree structure allows easy hand translation of the diagram into indented, fluent style code. We also have a tool that takes an XMind save file and gnerates the code from it automatically

While Xmind allows you to be creative in the beginning while you are still inventing domain abstractions (I couldn’t imagine doing without it), it is far from ideal once the abstractions have matured. At this point you are rapidly churning out user stories that are fairly obvious how to write, and Xmind’s limitations start to slow you down somewhat.

And even more recently, we have a purpose built graphical IDE for ALA. But it is not complete.

In my experience, a low overhead drawing tool is essential during the iteration zero design phase and during subsequent maintenance.

See the end of this chapter for an example project using Xmind.

We have used the word compose a lot so far in describing ALA. The term Composability means the ability to create an infinite variety of things by composing instances of a finite number of things. Composability is a very important property for dealing with complexity.

The Principle of Compositionality states:

The principle of compositionality restated for the context of software might be:

Brian Beckman, in his explanation of monads called "Don’t fear the monad" says that composability is the way to deal with complexity.

Jules Hedges says of this property "I claim that compositionality is extremely delicate, and that it is so powerful that it is worth going to extreme lengths to achieve it."

In software engineering, it is described by a pattern called "Abstract Interactions" or "Configurable Modularity" by Raoul de Campo and Nate Edwards - the ability to reuse independent components by changing their interconnections but not their internals. It is said that this characterises all successful reuse systems.

ALA has the property of composability by using domain abstractions with ports. The ports are instances of programming paradigms. The domain abstractions are the constituent expressions, and the programming paradigms are the rules used to combine them.

There are other software systems that have composability, usually using the dataflow programming paradigm, such as RX (Reactive Extensions), or more generally monads. Most composability systems are restricted to a single paradigm. In ALA, to achieve the correct level of expressiveness of requirements, multiple different programming paradigms are used.

In the above example, the user story code spans physical locations. I deliberately chose a Mars rover to make this physical reality as extreme as possible. There is obviously a lot of middleware infrastructure in-between to support the communications. And yet in the user story code we have gone ahead and wired the sensor directly to the display with only a conversion abstraction in-between. So how do those instances of abstractions get deployed?

Inside the user story abstraction, we can annotate the three instances with their physical locations. Another abstraction sits in a lower layer that knows about the concept of a physical view. It has already been configured to know about the three physical locations. The user story annotates the instances of abstractions it creates with where it wants them to physically run. It can do this because it knows about the physical view abstraction.

The physical view abstraction takes care of deploying the instances of abstractions for the user story to the correct locations, configuring them, and actually connecting them to the middleware. It also knows how to take care of version compatibility, and updating versions at different times at different locations.

The zero coupling of ALA, and zero data coupling in particular, allows all abstractions to be written by different parties or teams who don’t need to communicate with each other. They can even be written at completely different times. Only the knowledge dependencies must be communicated. In the example above, the user story abstraction cannot be completed without knowing about the three abstractions it composes.

The idea of no data coupling relies on a common programming paradigm. It relies on the teams who write the domain abstractions all using that programming paradigm. And it relies on having a separate team responsible for the user story, and all teams agreeing to use ALA and the common programming paradigm.

What happens if one of the abstractions to be used is written without knowledge of ALA? It has a conventional API that includes both configuration and data input/output methods. In this case the team responsible for the user story itself will write a wrapper that will make the abstraction into an ALA compatible domain abstraction that has a separate configuration interface and the relevant ports. The wrapper and the abstraction that it wraps become a single abstraction together.

If a system spans physical locatons, it is likely that the oranisation will allocate teams to develop code based where the code runs. Whether it’s frontend / backend or rover / lab, the teams are likely to be repsonsible for a deployed location. The teams then communicate with each other their APIs. It is highly unlikely that the teams will be told, in the interests of creating better abstractions, not to communicate with each other. And yet this is precisly what should be done to get the best architecture. Then there should be a team responsible for each user story. That team writes the cohesive user story code by composing using the set of domain abstractions provided by the other teams.

Unless these extra teams dedicated to user stories are put in place, the modules at different locations will end up with a lot of data coupling and collaboration coupling in general, just because the teams that write them will need to collaborate.

There will need to be be contracts that describe all this implicit coupling. The contracts will be a second source of truth, which must be kept updated.

The dependency injection pattern was introduced as an attempt to clean up the dependency mess created by OOP. It came too late to make the famous GOF patterns book. The authors wish they had included it instead of singleton. But dependency injection alone does not solve OOPs problems.

Previously we mentioned the use of dependency injection in ALA by using the wiring pattern to wire up instances of abstractions by their ports. The way this dependency injection is done is quite different to container based dependency injection.

Container based dependency injection works by matching interface types. The interfaces are implemented by one class, and required by another. The matching of this interface type is the implicit wiring of the two classes. There is no place where you can see the wiring explicitly. This is really bad. It is very difficult to trace the flow of a user story through the classes.

Now a class may be substitutable with another class that implements or provides the same interface. That’s why there is a container. You instantiate an object of the class you want to wire in, and put it into the container. But this is a far cry from general composability.

In ALA interfaces do not belong to the classes being wired. They are more abstract and represent a compositional concept which we call a programming paradigm. When a domain abstraction uses one of these abstract interfaces, either implementing it or using it, we call it a port. The abstraction has no implicit fixed arrangement with other abstractions. A separate abstraction in a higher layer is needed to specify how instances of abstractions with ports should be composed.

Note that this makes ALA not only fully polymorphic, but, in a way, extremely polymorphic. That’s because from the point of view inside an abstraction sending a message out of it port, there is potentially an infinite number of different abstraction types that it could go to. In conventional OOP, it’s typically a finite set, usually just a few. A finite set potentially allows the sender to have some implicit knowledge of those receivers.

Despite the extreme use of polymorphism in ALA, there is none of the usual disadvantage of indirection. Often indirection makes the flow of a program difficult to trace. But in ALA it is way easier to trace through a program. Consider it from two points of view. The first point of view is from inside an abstraction that is sending a message out via an output port. The abstraction doesn’t need or want to know where the message goes. That’s because along with extreme polymorphism comes zero coupling. You don’t need to know anything about the outside world to understand the sending code. The second point of view is outside the abstraction that is sending the message, the view of where it is wired to. Well that’s explicit and cohesive wiring to do with a given user story is in one place. You don’t have to trace through modules, you just reading normal code in one place in ALA.

Some abstractions are about describing change over time. ALA uses mutable state within such abstractions because these are abstractions that naturally contain state.

In functional programming, the way to express change over time is to pass the state into the function that represents the change over time, and the function returns the next state. The problem is it breaks the abstraction. Functional programming overcomes this problem by using monads. Monads can contain state, but are composed by pure functions. This hides the state in the more abstract and well tested monad itself. ALA uses this same principle. Domain abstractions completely hide their contained state in well tested code and are composed by pure functional code.

Monads vs domain abstraction is discussed in detail in chapter six. Here we are concerned with multithreading issues that can occur because we are using instances of domain abstractions that contain state.

We need multithreading strategies or rules to avoid the usual threading issues: race conditions, deadlocks or priority inversions. Unlike with conventional OOP programming, ALA’s zero coupling between abstractions makes it easier to put in place these strategies.

(In functional programming, the issues of multithreading are handled by using immutable data. I suspect that systems that only use immutable state also have problems when using multiple threads. For example one thread could be using and inconsistent or outdated copy of the current actual state.)

The strategies I use in ALA do not use locks. Making classes thread-safe with locks is highly problematic because insufficient use of locks results in race conditions, but liberal use of locks results in deadlocks or priority inversions. Successful locking requires detailed knowledge of the specific threading allocations of all objects. So if locks are implemented in the classes themselves it introduces coupling, destroying the abstractions. Locks would need to be implemented by the user story abstractions in the higher layer. That would encumber the user story code execution specific details.

Sometimes we need to use multiple CPUs for the shear amount of computing load, or we need to react to events faster than the single threaded model will allow. With a single threaded model, the maximum latency for the highest priority task is the time taken to process the longest of all tasks. This will be the longest compute bound task. In these performance cases we use either interrupts or multiple threads.

The ALA strategy is that synchronous message passing may only be used between instances of abstractions allocated to the same thread. Instances that are allocated to different threads must be wired using asynchronous message passing. As described in the previous section for the single threaded case, it is already easy in ALA to use either synchronous or asynchronous wiring for one way messages without changing the abstractions themselves.

Effectively this strategy is a knowledge dependency on an underlying convention. User stories must know that if they allocate two instances of domain abstractions to different threads, they must be wired using an asynchronous programming paradigm. With knowledge of the underlying convention, domain abstractions can be written without regard to thread safety.

You could argue that this knowledge dependency of user stories adds to the burden of the user stories. User story abstractions should just be about expressing requirements. However the user story is concerned with performance, because performance requirements are requirements too. Therefore, it is valid for user stories to do the allocating of instances to threads. The use of asynchronous message passing between them then follows.

This is the GALS principle (Globally Asynchronous, locally synchronous) applied to threads.

The fundamental ALA constraints could be applied directly to pure functional programming (not using monads). This would only require that the functions be good abstractions, and that functions would only call or use functions that are significantly more abstract than themselves.

But without monads, some functions would tend to be bad abstractions for two reasons.

1) Functions expose their inputs and outputs to the layer above, but the layer above is not interested in the data itself, only in the abstract concept of what the function does. It just wants to compose the functions with other functions, not deal with the run-time data.

2) Functions, or sets of functions, that naturally associate strongly with some state must have their state passed into and back out of them every time they are used. The layer above needs to handle this state data for the function. This is bad for the abstractions, both for the function and for the code in the higher layer.

+ Functional programming in one way encourages abstractions by not allowing side effects. However, this also destroys any abstractions that would otherwise be highly coherent with their own state.

+ This problem gets even worse when there are several layers of functions. The middle layer functions end up with extra parameters that don’t have anything to do with them, just so they can pass state data through to even lower functions. This makes these intermediate functions not great abstractions in themselves.

To solve these problems, functional programming uses monads. A full explanation of monads is in chapter six. This explanation is for programmers who are familiar with imperative programming in C#. They are compared in detail with ALA.

Domain abstractions together with programming paradigms are a more versatile, more powerful but conceptually simpler analogue of monads. Domain abstractions are a more general solution than monads that allows any programming paradigms to be supported, and arbitrary networks of communications between them. The fundamental idea of monads, that of separating execution details, state, and I/O into pretested units that are then composed using pure functional code is the same for ALA.

The ALA constraints can be used for either object oriented or purely functional programming. Either way, using abstractions that have zero coupling with one another changes how the code is organised.

In pure functional programming, if the data and the functions are separate good abstractions, then in ALA we would put both the data and the functions in the domain abstractions layer. Then user story functions in a higher layer would pass the data to the functions. However this handling of data by the higher layer functions, data that they themselves do not use, breaks them as good abstractions. Those top layer functions should only be concerned with composing the right data with the right functions, not handling the data itself.

Also, sometimes pure functions that require state to be passed into them are not good abstractions. An example would be an averaging filter for a stream of data. Passing the running average to the function every time there is new input data breaks an otherwise good abstraction. There are many situations when referential transparency breaks abstractions whose very nature is to describe change over time.

ALA prioritizes abstraction over referential transparency. Each approach is just a method to achieve analysability. Referential transparency attempts to improve analysability by always removing time from the analysis, even when time is a fundamental aspect of what is being described. It will expose implementation details if necessary to do it. In contrast, abstractions attempt to improve analysability by providing a simple learnable, zero-coupled concept, which in turn hides its implementation details completely.

Many systems or subsystems are inherently state machines - they are driven by events and need to change their behaviour according to their history. Programs for such systems ultimately have mutable state. Functional programming tends to separate that state out causing design-time coupling between functions and this state data.

State is a fundamental aspect of a computation. I define a 'computation' as some state and a function:

We can associate the state in two ways:

The first form is the pure functional form. The second form is the object oriented form.

In ALA we can use either form depending on where the abstractions are. When there is some state that is used by multiple user stories, then it would be put in its own abstraction. For example, a blackboard pattern or a game score could be an abstraction in itself. This abstraction could then be wired by user story abstractions in a higher layer to various abstractions implemented as pure functions.

In ALA we choose between these two philosophies on a case by case basis. When it makes sense to put state with methods as a class, and that makes a good abstraction, then there is no resulting coupling of that state with other classes. On the other hand, when it makes sense to keep state separate, we can put it in its own separate abstraction (in a class with ports). Either way, what you must not do is create direct associations between these classes. That will break them as abstractions. Instead, top layer user stories wire up instances of those classes by their ports.

In other words, the problem with object oriented programming is not that objects contain state per-se. It is that most classes are not good abstractions of a computation. We allow other objects to 'reach' into them or couple with them at design-time. They know there is state there, even if that 'reaching in' is via methods. This coupling is what makes conventional programs with state so hard to reason about.

In summary, ALA works by preferring abstraction over referential transparency.

In chapter six, there is an explanation of monads in terms of concrete code for people familiar with imperative programming. Here we assuming prior familiarity with monads in comparing them with ALA. ALA includes the power of monads, but in a more powerful, more flexible and more straight-forward way that can be applied to all parts of the code.

When we use the term monad library in this section, we are referring to a whole library based on monads. For the most common monad, which is the monad based on the IEnumerable interface, this library consists of the familiar functions such as Select, Where and Order found in LINQ.

Strictly speaking, the monad itself only includes the Bind function (which is SelectMany in the case of the IEnumerable monad), not all the other map and reduce type functions you find in the library as well. A monad actually has three components, A Type<T> that it works on such as IEmuerable<T>, the bind function, and a function to convert a T to the Type<T>.

Deferred/push monads, as described above, are closest to the preferred ALA dataflow programming paradigm. Deferred/push monads have two useful properties.

One of the core tenets of ALA (as discussed in an earlier section) is "Composition using layers" instead of "Hierarchical decomposition using encapsulation". Unfortunately Ndepend is designed with the assumption that the application should be built using the latter approach. It likes to present a decomposition structure, starting with assemblies (packages) at the outermost level, then namespaces, and then classes. I’m not sure why it considers namespaces a viable encapsulation mechanism because they don’t provide encapsulation.

Namespaces provide no useful decomposition structure. They do not make abstractions in themselves, nor do they implement a facade pattern or an aggregate root type of pattern with even logical encapsulation. Any classes inside each namespace can have unconstrained direct relationships with any classes in any other namespace.

So Ndepend out of the box gives us a false picture, because it omits all dependencies that go in or out of namespaces. To really get an idea of what the class dependency graph looks like, I configured Ndepend to use a query that gives me all the classes in all the namespaces. Here is what the legacy application truly looks like:

This graph is very large. Right click on it, and open it in a new tab in your browser, so you can zoom in to see the dependencies in the background. It is truly frightening. Ndepend had no chance to find the dependency layers. There may be vague onion type layers going outwards from the middle. It makes readily visible why continued maintenance on this application is so difficult. You have to read a lot of code to find even a tiny part of this hidden structure.

The developer who maintains the application tells me this is a fair reflection of the complexity that he has to deal with.

To be fair, some of the dependencies in this diagram are 'good' dependencies (as described in an earlier section on good and bad dependencies). For example, the box near south-east called ScpProtocolManager has a lot of dependencies coming into it, which means it is possibly used a lot and therefore is potentially good abstraction. Ndepend does not know about the concept of good and bad dependencies, but if it did I would have it just display the bad ones.

Here is the equivalent Ndepend generated class dependency graph for the new ALA version of the application. This graph has the classes from all namespaces.

You can see the three ALA layers which are vertical and go from left to right. Only the Application sits in the top layer. The DomainAbstractions layer contains the next two columns of classes and a few from the next column. And the ProgrammingParadigms layer contains the rest on the right. Actually there were a couple of bad dependencies present when this graph was generated which have since been fixed. (There should be no dependency between Panel and OptionBox, nor between Wizard and WizardItem.) With these removed, the graph would form into the three abstraction layers.

The newly rewritten application is a work in progress at this point. However, as features are added, this is all the dependencies you will ever see. The Application already uses most of the domain abstractions we will ever need, and the domain abstractions already use the programming paradigm interfaces they need. There are a few DomainAbstractions to be added, but this is essentially what the class dependency graph will stay looking like.

As of this writing, the new ALA version of the application is still a research project, but so far everything has gone smoothly with two weeks spent doing the description of the requirements as a diagram, and 18 months so far spent writing the domain abstractions. So far there are no issues getting it to actually execute. It is expected that we will actually commercialize the project soon and replace the old application.

As we said in this chapter, diagrams can be an important aspect of ALA when the user story naturally contains a network of relationships among its instances of abstractions. In this application this is the case. There are UI relationships between elements of the UI. There are dataflow relationships between UI elements, data processing elements, and data sources. There are event-flows from UI to wizards and between wizards and the SaveFileBrowser. and there are minor dataflows such as a the filepath from the file browser to the csvFileReaderWriter.

Here is a sample section from the application diagram that shows all the relationships that implement the user story:

This diagram was drawn using Xmind. It shows a single user story. There is a UI with a menu item or a tool bar to start the user story. It then displays a browse dialogue to specify the location of the file. When the filepath has been selected, it gets data off a device on a COM port, using a protocol, and writes it to a CSV file. The data is also routed to be shown on a grid on the UI.

The user story diagram makes use of four different programming paradigms (which become four different interface types). Firstly there is the UI structure consisting of the window with its menubar, grid etc arranged inside it. Secondly, there is an event connection for when the menu is clicked which opens the browse dialog. Thirdly a dataflow connection carries the output of the browse dialog, a string containing the selected filepath, to the CSVFileReaderWriter. Another dataflow connection carries characters between the COM port and the SCPProtocol and another carries SCPcommands from the SessionDataSCP. The forth programming paradigm is a table dataflow that carries dynamic columns and rows of data from the SessionDataSCP object to the grid object in the UI and to the CSVFileReaderWriter.

Having drawn the diagram to represent the user story, we need to make the diagram execute. When we started this particular project we had no tool for automatically generating the code from the diagram, but during the project, one of the interns wrote a tool to do this. It parsed the Json output from Xmind and generated C# wiring code equivalent to what we will show below.

However, at first we were hand generating code, and it is instructive to know what this hand generated code looks like, just so we know how the diagram actually executes.

When we were hand generating the code, it was important that the code was readable from the point of view of seeing how it corresponds exactly with the diagram. (It wasn’t important that the code was readable from the point of view of seeing how the user story works - that was the job of the diagram.) We had various conventions to support the one to one matching of diagram and code. One of these conventions was to indent the code to exactly mirror the tree structures in the diagram. Another was that whenever a new instance of an abstraction instantiated, all its ports would be wired immediately, and they would be wired in the order they were declared in the abstraction. This implies a depth first wiring strategy, analogous to walking the diagram tree depth first. Any ports with cross connections (the red lines in the diagram) would also be wired to their destinations at the time the abstraction were instantiated. If the destination instance did not already exist it would be pre-instantiated.

Using these conventions, it is a simple matter to hand generate the code below from the diagram.

We used a 'diagram first' rule to keep the diagram and code in sync. Change the diagram first, then change the wiring code.

As of this writing, a graphical IDE is being developed for these types of ALA applications.

The interface for synchronous events was given just above.

The difference between this and your everyday common imperative function call or method call is only the indirection. The sender doesn’t know who it’s calling. (In conventional code, indirection creates problems, but in ALA these problems do not exist, so all communications use this type of indirection. This is discussed further later.)

First let’s create two dummy domain abstractions with ports using the synchronous event driven programming paradigm. The two domain abstractions will use and implement this interface respectively:

Now we can write an application that wires an instance of A to an instance of B.

The output of the program is "123".

The Main function instantiates one instance of each of our domain abstractions, and wires them together. (If you have not seen the WireTo abstraction before, it is an extension method that uses reflection to search in class A for a private variable with a type that is an interface. It then sets it pointing to the instance of B if B implements the interface. WireTo is not central to the current discussion, the IEvent interface is. WireTo is discussed in more detail in the example projects of chapters two and three.)

Notice just how abstract IEvent is. It’s highly reusable. It’s not specific to any domain abstraction or the application. It just knows how to transmit/receive an event. Because it is so abstract, it is stable. The more domain abstractions that depend on it the better, as that will allow them to be wired together in arbitrary ways, which gives us composability.

The IEvent interface can be compared with the observer pattern (publish/subscribe) which also claims to achieve decoupling. However the observer pattern only reverses the dependency of a normal method call. Instead of the sender knowing about the receiver, the receiver knows about the sender (when it registers for the event). If the sender and receivers are peers in the same layer, the observer pattern does not solve the problem. The IEvent interface decouples in both directions. The job of 'subscribing' is moved to the application layer, because only the application should have the knowledge of what should be wired with what.

In the above example, we used the word event, but implemented it in a specific way (a synchronous method call). The terms event and event driven may have overloaded meanings. To some it may mean asynchronous or it may mean observer pattern (an event is a public thing you can subscribe to), or it may mean both.

In ALA the term means neither of these. As a programming paradigm it simply means that we think of programming as reacting to what happens instead of prescribing what will happen next - a reactive rather than prescriptive programming style. They can be either synchronous or asynchronous. They are never public - the layer above always wires them up from point to point explicitly. Events can be wired to fan-in or fan-out.

We discuss the meaning of synchronous and asynchronous in more depth later, but here we just want to see how asynchronous can be implemented at the code level. Synchronous and asynchronous are two different execution models for the same programming paradigm.

To implement the asynchronous execution model, conventional code may use an event loop that works something like this: the originator of the event calls a Send method on an EventLoop object. It passes a reference to a function or method of another object that it wants to send the event to. The Send method in EventLoop creates an object that represents the event and puts it into a queue. The Send method then returns. The main loop resides in this EventLoop object. It loops taking events from the queue one at a time and calls the referenced function or method. This is sometimes called the reactor pattern, but its actually a simplified version of reactor so we will call it simply an event loop.

For ALA, the only difference is that the sender can not specify the receiver function and object.

Here is the application layer code:

The difference with our previous synchronous application is that we first spin up an asynchronous execution engine object called eventLoop. The WireTo is used in the same way except that we pass in the execution model.

Here are the A and B dummy domain abstractions again. They are identical to the ones we used for the synchronous version above.

When this program runs, it will print "132" instead of the "123" that the synchronous version did. At the domain abstraction level, we need to not care whether it is "123", or "132". If we do care, then we need to use a different programming paradigm.

Now let’s have a look at the programming paradigm abstraction to see how it works.

This mechanism of overloading the WireTo method can be used by any programming paradigm.

The WireTo method instantiates an intermediary object, stores the receiver object cast as the interface into it (which is effectively wiring the intermediary to the receiver), and then calls the standard WireTo in the Foundation layer to wire the sender to the intermediary object.

In this example we have put the main loop inside the execution model for simplicity. We would not normally do this because we may have several different programming paradigms each with their own main loops. So we could have the main loop in the Foundation layer, and the different execution models would register a Poll method on it. Alternatively we could make the loop function an async function that awaits on an awaitable queue implementation.

As usual in ALA, we do not try to decouple anything inside the AsynchronousEvent.c abstraction. Everything in it cohesively works together.

The propose of the examples is to show that we can create our own programming paradigms and that their implementation can be simple.

In conventional code, communications generally follow dependencies directly. If we try to think in terms of layers, with dependencies always going down the layers, these communications always go either up or down. So we may not be used to thinking of sideways communications. Or if we do allow sideways dependencies within a layer, we may not be used to thinking about sideways communications and up/down communications as different things.

In ALA, we need to think of them differently. Let’s refer to sideways communications as wired-communications, and up/down communications as abstraction-use-communications.

A common example of abstraction-use communication is when you configure an instance of an abstraction by passing parameters to the constructor, or by calling setters. Another example is calling a squareroot function in your math library. A common example of upward communication using abstraction-use-communication is executing a lambda expression that has previously been passed in to an instance of an abstraction during its configuration. Upward calls are always indirect in some way, such as the mentioned lambda expression, passing in an anonymous or named function, observer pattern (publish subscribe), callback, or strategy pattern. We don’t use virtual functions in ALA for up calling because we don’t need or want to use inheritance.

A common example of wired-communications is when an instance of an abstracton sends something out on a port. It arrives at the input port of another instance of an abstraction to which it was wired by the layer above.

In all the following discussions of programming paradigms, we will be talking about wired-communications unless noted otherwise. Note that we use the word communications to cover for both events and dataflow types of programming paradigms. Another common term is message.

Sideways communications in ALA is always indirect. The sender never names the receiver or the function or method in that receiver. Conversely, receivers never register themselves to a sender, or to a public event. Global event names are never used. Communications always follows the wiring put in place by the layer above.

In conventional code, there is a downside associated with indirection, which is tha it becomes harder to follow the flow of execution through the modules for a given user story. That downside does not exist for ALA. In fact it is the opposite - it is easier to trace the flow of calls through the system. This is because user stories are expressed in one place cohesively. You see all the explicit wiring of a user story abstraction in one place instead of tracing it through multiple modules. Only if an abstraction it uses does something unexpected do you need to drop down inside the abstraction, and enter a different self-contained self-cohesive set of code.

When reading code inside an abstraction, it is in the very nature of abstractions that they know nothing of the outside world. They do not need or want to know where events come from or go to externally. Indirection is used so that flow can lift out of the internals of an abstraction to the more specific wiring code in a layer above.

Even synchronous function calls are always indirect. At run-time, the inside of one abstraction synchronously calls a method inside another abstraction under the direction of the wiring in the layer above. But at design-tie, it has no knowledge of what that other abstraction or method is. Whether the run-time execution model is synchronous or asynchronous, push or pull, with fan-out or fan-in, the wiring model between instances of abstractions with ports is always indirect.

Even if the communications is asynchronous, the caller does not send the event to a particular destination, nor does it give the event a global name so that receivers can register to it. Both patterns would involve a bad dependency. Instead it still only goes as far as its own output port.

Conventional code will often use an interface or the observer pattern (publish-subscribe) (or C# events) to invert a dependency. If the two modules were peers in the same layer, inverting the dependency by adding an indirection only makes the program even more difficult to follow. ALA does not need to use the dependency inversion principle or the observer pattern for peer to peer communications because there is no dependency. In other words ALA completely sidesteps the dependency inversion principle and the observer pattern for all communications between peers.

ALA generally uses dependency injection directed by explicit wiring.

Having said that ALA does not use the observer pattern (or any other form of the receiver subscribing to senders in the same layer), the observer pattern is sometimes used within a programming paradigm interface. Consider a programming paradigm where communications is needed in both directions. In the same direction as the wiring, it is usually implemented as a simple method call. The way interfaces work in our programming languages, the A end uses the interface and the B end implements it. The asymmetry is a shame. If we want a method call in the other direction, we use the observer pattern inside the interface. The publisher, the B end, implements the observer pattern. The subscriber, the A end, subscribes to it. The difference from the standard observer pattern is that the subscriber does not know the publisher. It is only subscribing to it indirectly via the interface.

If a dependency were going up from one abstraction layer to a higher one, then of course we invert the dependency. But a dependency from a more abstract abstraction to a more specific one doesn’t make sense in the first place and so this situation never occurs. The dependency inversion principle is already built into the ALA constraints, so you never need to invert dependencies later.

If we are using standard synchronous function calls or method calls as the execution model, we have a choice between push and pull. In other words, does the sender of an event or data initiate the call, or does the receiver?

Push

Pull

In conventional code, the decision as to whether to use push or pull is often dictated by the need to control the direction of dependencies. To change a pull to a push without reversing the dependency would require indirection or the observer pattern. Similarly, to change a push to a pull without reversing the dependency would require an indirection. So usually we use the one that allows us to use a simple function call with the dependency in the desired direction.

With ALA, most run-time communications take place within a layer, and there are no dependendencies between the abstractions involved. Instances of abstractions are wired using interfaces that represent programming paradigms:

Because there are no dependency constraints, we are free to choose between push and pull. Usually it would be for performance reasons. If the source data changes infrequently we would use push. If source data changes frequently, and the receiver only needs the value infrequently, we could choose to use pull.

It would be nice if you could choose between push and pull at wiring time. In other words, we design domain abstraction ports to handle both push or pull, and you choose push or pull when wiring instances in the application. For example a signal filter could support both push and pull. If not we might need two version of the filter. Unfortunately it increases the amount of code inside the abstractions. So we usually write abstraction ports to use either push or pull.

To allow optimal composability of abstractions, I use push ports by default so that most ports can be wired directly. Push also works quite naturally for events. It means that the initiator of an event pushes it as soon as it happens. The opposite is possible: receivers poll the source when they are interested to know if an event has occurred.

For dataflows, push means that the data 'flows' whenever it changes. This works better performance-wise if the data does not change too frequently. It works well when all data must be processed. It is ok when all the data does not need to be processed, and only the latest data is important. Push is usually more efficient than periodically polling for data.

A final factor in the preference to use push by default is that push ports can be wired for either synchronous or asynchronous execution models without changing the domain abstractions (discussed above in the section on synchronous vs asynchronous). To allow this for pull ports requires the pull end to be written for an asynchronous execution model, which can be awkward. This aspect is discussed more fully in the section on the request/response programming paradigm later.

For all the above reasons we use push ports by default, and pull ports when we have to. It is analogous to using RX (reactive extensions).

Remember we are talking about 1-way communcations. In a later section we discuss programming paradigms that use 2-way communications.

In chapter three, we used the terms fan-in and fan-out in relation to dependencies down layers. Here the terms fan-in and fan-out are used for something completely different. Here we are talking about wiring.

Fan-out means that an output port of one instance of an abstraction is wired to many instances. Fan-in means many instances are wired to a single input port. It depends on what makes sense for each particular programming paradigm.

C# and other languages don’t allow an interface to be implemented more than once. Sine we use interfaces as ports in ALA, this can be a serious limitation.

For example, consider implementing an AND gate with 4 inputs all IDataFlow<bool>.

Implementing IDataFlow<bool> more than once like that gives a compiler error.

It’s a valid thing to do however. I can only assume that outside of ALA, no one seems to have needed it. In fact the whole concept of ports should be part of all object oriented languages. Only then would OOP realize it’s potential for reuse. (ALA is really just OOP done right.)

If the C# language allowed the same interface to be implemented multiple times, the only syntactical difference would be that the implementations would be given names:

You would be able to set a reference to the object’s interface using this name instead of casting to the interface type.

Java almost allows this to be done using method references. But it only works when there is one method in the interface.

We already used a work-around for this limitation of C# in the Add domain abstraction in chapter 2. In that work-around we created a Double2 type which was a simple struct containing a double. That allowed us to implement both IDataFlow<double> and IDataFlow<Double2>. But it’s not a general solution.

A more general work-around for this limitation of C# is to use interface fields instead of interface implementations and reverse the wiring.

We append an "_R" to the name of the interface to indicate it is a 'reversed wired' interface. Here is the interface:

The receiver registers an event handler method to the event in the interface:

To complete the workaround we need an intermediary object. Both the sender and receiver are wired to this object. It implements both IDataFlow<T> and IDataFlow_R<T>. The class for this object resides inside the IDataFlow programming paradigm abstraction:

The above code is also added to the DataFlowFanout class listed above. See note 3 in that listing. This allows the DataFlowFanout intermediary object to be used for the purpose of this workaround among its other uses.

A problem with this workaround is that you need to wire in the reverse direction to the flow of data. So if data is to flow from A to B, we would need to write:

This is unintuitive at the wiring level.

We would prefer to write like we do normally:

We can write an override of WireTo in the programming paradigm abstraction and register it with the Foundation WireTo.

The override WireTo would look for a field interface in A that matches a field interface in B by name with a _R suffix.

TBD write the override WireTo.

To accomplish wiring, the application, feature or user story abstraction’s code makes calls to the WireTo method, passing in the two object/ports to be wired. The WireTo method, by default, wires the two objects by assigning the second object to a private field in the first object, provided the interface matches. This default behaviour sets up a direct connection between two communicating objects.

For arbitrary execution models, we don’t always want direct connection between connected objects. We may want an intermediate object to be automatically wired in, or other special behaviours. For example if the two objects being wired are in different locations, we will want to automatically wire in the necessary middleware intermediary objects. Intermediary objects are commonly needed in ALA. We have previously used them for several different purposes, such as asynchronous communications, pull communications, etc.

In the asynchronous programming code earlier in this chapter, we used an override of the WireTo method that had an extra parameter. But what if there is not extra parameter. Then the WireTo method that resides in the ALA foundation layer is the one that will be called. It can’t know anything about programming paradigms or execution models in higher layers. But it can know in an abstract way about allowing itself to be overridden.

The WireTo method in the foundation layer can support a list of registered override functions. It calls every override function in the list. If all return false, then it does its default behaviour.

The foundation WireTo can first do the reflection work. It can create lists of potential field and implemented interface ports in both the A and B objects. Then it can pass these lists to the override functions.

TBD Modify WireTo to support run-time overridding. Use it to implement a null decorator intermediary on the synchronous programming paradigm. Then use it to implement wiring of a push port to a pull port and a synchronousmiddleware for wiring objects in different locations.

Consider a wiring topology of an application in which wiring diverges from a single instance of an abstraction, and then converges to a single instance of an abstraction. The two paths will be executed at sightly different times. So one input of the end instance will get data from the common source before the other. During the time between the two, the inputs may be in an invalid state. This is what we mean by a glitch.

Glitches also happen in conventional code where they are a cross cutting concern. They also even happen in electronic circuits.

In ALA, they are a concern within a single abstraction, either the application abstraction, or a feature or user story abstraction. This is where the diamond topology of the wiring is apparent and the problem can be easily understood.

Abstractions may have a minor inputs which it expects to get data first and major input that triggers operation. In such a scenario, the application can control the order of execution in the wiring so that the major input gets its data last.

One solution is to provide an trigger event port on abstractions that have multiple inputs. The application must trigger the port once all inputs are valid.

It is a future topic of research to automatically detect glitches on abstractions with multiple inputs, and potentially to automatically resolve diamond wiring glitches.

In ALA, it is no problem to have circular data paths. Note that by circular, we are referring to wiring inside an abstraction, not dependencies between layers. Circular wiring naturally occurs in feedback systems, just as it does in electronics. It is nice to be able to represent such feedback systems directly in the wiring.

In conventional code, circular data paths may need a pull or an indirection to avoid circular dependencies. ALA does not have this problem. Circular wiring is as natural as it is in electronics.

A programming paradigm’s execution model needs to consider circular wiring. For example, circular wiring using all synchronous programming paradigm will result in an infinite loop at run-time, just as it does in conventional code. It easy to solve however. It can be as simple as an abstraction instance placed in the circuit that does an asynchronous call, or an abstraction instance that does a delay. This effectively causes a return to the main loop where the circuit can be called again. The main loop can process higher priority tasks first. It is no problem for such a circuit to repeat forever.

Alternatively, we can implement programming paradigms utilizing existing rigorous execution models, such as the discrete time execution models used in function blocks or clocked-synchronous execution models. The continuous time execution model underlying Functional Reactive Programming will automatically flags such loops.

While circular data loops can occur in conventional code as well (recursion), they are more likely in ALA because ALA is likely to have dataflow abstractions which can easily be wired as a circuit. However, in ALA it is usually explicit and clear in the wiring diagram or code. The Calculator project in chapter two contained Dataflow loop circuits.

Although we already did simple coding examples for synchronous and asynchronous execution models at the start of this chapter, the design choice between synchronous and asynchronous needs deeper considerations.

Synchronous communication is like asking someone a question. You stop your life and wait, albeit for a brief time. You don’t resume your life until you get the answer or a nod. Asynchronous communication is like sending an e-mail.

Synchronous means that the calling code resumes execution after the callee has finished processing the communication.

There are reasons why you may want to use synchronous communications. The communication may cause a side effect, which we want to be sure is completed before continuing execution.

If the receiver will take a long time to execute, which can be for many reasons such as a long running algorithm, receiver not ready, external IO, a deliberate delay, etc, then a synchronous call will do what is referred to as blocking. Blocking means the thread will stop and wait. If the blocked thread needs to do something else in the meantime, this blocking will be a problem in one way or another.

In ALA we prefer single threaded solutions. Multi-threaded programming should only be used for performance reasons e.g. meeting a challenging latency or throughput requirement. A single threaded system will use run to completion, so in that respect is commonly referred to as cooperative. Being cooperative sounds like it doesn’t comply with ALA’s zero coupling. To some extent this is true, but the requirement to keep all routines short (non-blocking) can be thought of as an abstract requirement from a lower layer rather than relative coupling between domain abstractions. All higher abstractions need to know about this. Usually if nothing in an application blocks, the latencies needed for an application to respond to a human in reasonable time (which is the most common soft deadline requirement) will be acceptable. Using a single thread when things take time, or things need to happen in real time requires asynchronous communications.

Asynchronous means that the sender instance’s call returns before the callee has finished processing the communication. It will usually be before the callee even receives the communication.

Asynchronous calls can be implemented in several different ways. What they all have in common is that the caller makes a synchronous call that starts the communication or starts the callee’s execution in some way. The caller will then resume executing the next line of code pretty much immediately.

In ALA, as with the synchronous case, the caller does not know where it is sending the communication and the callee does not know where it came from. Where synchronous and asynchronous communications differ is only in when the call returns.

Note that here we are discussing the fundamental case of one way communication. We will consider two way communication programming paradigms later.

With one way communication, we have the option to decide at wiring time whether to use synchronous or asynchronous, provided the sender doesn’t care whether it resumes processing before or after the receiver gets or processes the message.

Some common ways of implementing asynchronous calls are:

Other mechanisms are possible. Note that all of these mechanisms describe how the sender’s synchronous call returns before the receiver completes.

Remember that in all these implementation examples, we are talking about fundamental one-way communication - an event or pushing some data. Two-way communications gets more complicated, and is discussed below.

All the asynchronous programming paradigm execution models discussed above use pushing. Analogous pulling asynchronous communications are also possible. For an asynchronous pull, the receiver makes a synchronous call which returns a previously calculated result without waiting for the sender to calculate it. It returns the last result available from the sender, or a value from a FIFO, etc. The sender will calculate new values in its own time.

Asynchronous communications has inherent concurrency. This simply means that tasks of different features or user stories or channels or whatever can be executing in an interleaved fashion. That’s why we are using it. The concurrency is at a courser grained level compared with pre-emptive multitasking. There can still be a need to lock any resources that can be in an invalid state for a time, or to think in terms of transactions.

Generally ALA can use both asynchronous and synchronous execution models in its programming paradigms. It does not have rules for when to use one or the other. The design choices remain more or less the same as in non-ALA applications according to real-time factors discussed above.

However, ALA is all about abstractions and zero coupling at design-time. It would be good if the abstraction didn’t need to know whether the external communications beyond their ports is going to be asynchronous or synchronous. We would like to decide that when we wire instances of them up. It is therefore desirable that domain abstraction ports that generate events and ones that listen to events can be wired for either synchronous or asynchronous execution. That way, for example, they can be wired synchronously by default for best efficiency, but asynchronously if they are in different locations, or if the recipient will take a long time.

Synchronous communications are deterministic. They prescribe the order in which everything happens. Furthermore, they effectively put a system wide lock on everything until the entire function calling tree completes. Nothing else can happen anywhere until it finishes.

Asynchronous communications, on the other hand, is inherently less deterministic. The non-determinism is made necessary by the external system, things like: real-time I/O, external networks, or by the need to improve performance.

During asynchronous communications, the functions can be executed in the order in which they are scheduled (using a simple queue) by default. This is what we did in the sample code at the start of this chapter. If this ordering scheme is used, then from the point of view of an asynchronous call tree, the natural order of execution is different from the synchronous function call tree. A synchronous function call tree will be depth first, whereas an asynchronous function call tree will be width first.

During the execution of a call tree, other call trees may be executing in parallel. This does not mean parallel in the fine grained sense of multithreading. It means parallel in the course grained sense that between the execution of asynchronous functions, other functions of other call trees may execute.

One consequence of asynchronous communications is that if any resource, including any object, is left in an invalid state between the running of two asynchronous functions, it must be locked. The need for locking is much less common than in a multithreaded situation. How locking can be accomplished without introducing coupling into the abstractions by using an arbitration programming paradigm is discussed later. Locking will change the order that functions execute.

The order of execution of asynchronous functions, can also be explicitly changed using priorities. Priorities are usually used to explicitly improve performance by doing more urgent things first.

Because the order of execution is outside the control of the abstractions involved, domain abstractions should not care about when it’s one-way asynchronous communications are executed. If the priority system were to reverse the order of execution of every asynchronous function in the system, a domain abstraction results should be the same (except for its performance). If the order does matter, the order needs to be explicit in some way. For example, a domain abstraction could use a 2-way communications port so that it gets a communication back when something is complete and it can move onto the next step. Another example is to use an activity programming paradigm (UML activity diagram). Abstractions have start input port and a finished output port. The application wires instances of them in a sequence.

Priorities are generally a system wide concern, so the application abstraction (or feature or user story abstractions) are the only ones that have the knowledge to know how to set priorities.

In conventional modular systems, priorities are usually a cross-cutting concern, but in ALA they are cohesive with the wiring code, which is already in one place for a given feature or user story. The application may need to. prioritize the features and user stories.

TBD Show example implementation code for priorities. Add an optional priority parameter to the WireTo of the asynchronous programming paradigm abstraction. The appplication can use priority numbers such as 0,1,2. We need a default priority so that WireTo can be called without specifying a priority. The application would configure the default, for example to 1.

The final requirement is that applications can still use the asynchronous programming paradigm without using priorities at all. There would be a default default priority level of 0.

A priority abstraction could be created in the domain abstractions layer. It would contain a dictionary for priority levels. You would not use an enum for priority level in this abstractions because the levels are specific to an application. The application configures the dictionary with level names such as Low, Middle, High, that associate with numeric priorities.

Also, we need to consider if domain abstractions may ever want to use priorities internally. If so we need to do it in such a way that they do not have a dependency on a priority abstraction because if they are used without priorities, we don’t want to have to include the priority abstraction.

When a resource that takes is used asynchronously, more than one user may try to use it at the same time. For example a transaction on a database may involve several asynchronous function calls, and have multiple users. It would need to be locked for the duration of the transaction. Or an ADC converter that takes time to do a conversion may be used by multiple users. It would be busy to new requests while it is performing a conversion. If the resource is busy, the communication to the resource will need to be queued until it is ready.

The reactor pattern can handle this situation. It can check if the receiver is busy before giving it the communication. The dispatcher wont remove asynchronous function calls from the queue unless the destination resource is ready for it.

If a simple event loop is used, a solution to this problem, is an intermediary object that is wired in front of the resource. It keeps its own queue of event objects. When the resource signals that it is free, it takes the first event from the queue and sends it to the resource via the main event loop. That way only one event at a time can be in the event loop’s queue.

As discussed in earlier sections, synchronous and asynchronous 2-way communications have different advantages. There is a principle, GALS, that suggests that we use synchronous locally (within a processor) and asynchronous globally across processors. I think this is too simplistic. There are reasons other than cross-processor communications that cause certain communications to take real time, such as IO or delays. These communications should be asynchronous, and then all the ones that might be wired to them need to be asynchronous as well. Nevertheless there may be some communications in the average application which needs the advantages of synchronous communications.

The request/response ports of domain abstractions may end up a mixture of synchronous and asynchronous.

If the requester is asynchronous and the responder is synchronous, there is little problem in connecting them using an intermediary object. When the requester calls the intermediary, the intermediary in turn calls the responder which returns immediately. The intermediary then places the result in the task or future object, or calls the requester back if it uses a callback.

If the requester is synchronous and the responder is asynchronous, it would be possible to create an intermediary adapter, but it will block the requester’s thread, which probably isn’t what we want. The requester would need its own thread (or its requester), which, as I said earlier I don’t recommend as the way to solve this problem. So they are essentially incompatible. The requester code would need to change to asynchronous, as described by one of the methods above.

Another section of this chapter discusses the dataflow programming paradigm. Dataflow can be similar to event-driven with a parameter when it pushes data. However Dataflow has variants where can be synchronous, can be pull rather than push, and can send a whole table of data in batches. For this reason Dataflow and event-driven are considered different programming paradigms.

Event-driven programming is a reactive style in that it contrasts with the prescriptive or orchestrated style of the imperative or activity programming paradigms. In event-driven, the system is idle until something happens, and then things react to it, possibly changing some state, possibly generating more events, completion events, or timeout events. Event driven systems like to use interrupt routines to get events from the outside into the system. The interrupt routine puts the event directly into the main loop event queue.

In a reactive system, we don’t know what will happen next, in either the outside world or what code will execute next. It is less deterministic. Reacting to an event often changes some stored state. This state may change the way we will react to subsequent events. In other words, event-driven often goes hand in hand with state machines.

Event-driven programming is generally not thought of as a request/response type of paradigm. There can be a response, but it would be thought of as a completely separate message that needs its own point to point wiring. We don’t need synchronous communications because there is no response associated with an event (in the same wiring).

ALA is polyglot with respect to programming paradigms, so there is no reason to try to make an entire system either event-driven or prescriptive. Both can be mixed for maximum expressibility of the requirements.

When there are no forces favouring reactive or prescriptive, I generally default to reactive. This is because reactive systems are more versatile in maintenance. A prescriptive style becomes awkward when an unforeseen event needs to be handled in the middle of a prescriptive routine. The flow of the routine becomes more complicated.

Some conventional event-driven systems use global event names for inter-communication between modules. Each receiver names the events it is interested in, effectively a variation of the observer or publish subscribe pattern. They do this by registering to global event or signal names. This is considered relatively decoupled by its proponents, because senders and receivers don’t know directly about each other, only about global events names. It is illegal in ALA because most events will not abstract enough to be named and become globals. They will tend to be specific to pairs of modules that need to communicate.

Event names then essentially become symbolic wirings. Symbolic wiring is difficult to follow because you have to search for where the names appear throughout the entire code.

By effectively collaborating on symbol names, abstractions are coupled with each other still. It’s a rigid system because modules could not be rewired in a different way without changing them.

In ALA we use point to point wiring instead, or should I say port to port. Wiring is brought out to a coherent place. Because the wiring is point to point, the events are anonymous. You don’t have to name the lines on a diagram.

Having said that, it is possible to have an event that is abstract enough to go into a layer below. Such an event would need to be used by many many domain abstractions so that it is truly more abstract. If only a few domain abstractions need to use an event, then they should still use ports and be all wired up.

If you do create a global abstract event, it would be so ubiquitous that you never want to use the domain abstractions without it. They will have a dependency on it after all.

I can’t think of an example of such an event. Perhaps an event called initialize. It is generated after the wiring code has executed but before an application is set running. Domain abstractions use it to do initialization that needs the wiring in place.

Another example may be a closing event, giving domain abstraction instances a chance to persist their context data before the application closes down.

I frequently use dataflow execution models.

Here is one variation which works well:

TBD

This variation has these properties:

Here is the version I use most often.

TBD

Note that domain abstractions may not collaborate on a specific type for T. A pair of domain abstraction may not, for example, share a DTO (data transfer object) class as that would then be an interface specific to one or other of those classes. T must be more abstract and come from a lower layer, so is often a primitive type from the programming language. T may be passed in by the application, which always knows types of data moving through the system.

Type inferencing is desirable. For example, an instance of a DataStore<T> abstraction could be configured by the application to have some specific fields. Ideally this is the only time the application specifies the fields. The application wires it to a select abstraction that removes one field and then to a join abstraction that adds one field. From there it is wired to a form abstraction that displays the fields. Ideally the form, select and join abstractions do not also have to be configured by the application to know the types of their ports. Instead they are able to infer the type as an anonymous class as it goes from port to port at compile-time.

This interface moves a whole table of data at once. The table has rows and columns. The columns are determined at runtime by the source.

Run-time types can also be used. For example, the fields in an instance of a table abstraction may not be fully known at compile-time. This is especially true if the table abstraction provides persistence, or, for example, if the data source is a CSV file with unknown fields. In this case a ITable programming paradigm would transfer type information at run-time as well as the data itself.

TBD implementation examples

All systems can have glitches when data flows are pushed in a diamond pattern. The diamond pattern occurs when an output is wired to two or more places, and then the outputs of those places eventually come back together. If they never come together, even both seen by a human, then we generally don’t care what order everything is executed in. But when they come together, the first input that arrives with new data will cause processing, and use old data on the other inputs. This unplanned combination of potentially inconsistent data processed together is a glitch. It even happens in electronic circuits.

The following composition of dataflow operators is meant to calculate (X+1)*(X+2)

When X changes, there can be a glitch, a short period of time, in which the output is (C_new+1)*(C_old+2).

In imperative programming, this problem is up to the developer to manage. He will usually arrange the order of execution and arrange for a single function or method to be called at the place where the data-paths come back together. As he does this, he is introducing a lot of non-obvious coupling indisde the modules of the system, which is one of the big problems with imperative programming.

When we have composability, we don’t know inside the abstractions how data will propagate outside, and how it will arrive at its inputs. We want to execute whenever any of our inputs change, because as far as we know it may be the only change that might happen. So we really want the execution model to take care of eliminating glitches automatically for us.

This is a work in progress for the IDataFlow execution model described above. In the meantime, as a work-around I take care of it at the application level using a pattern. When I know dataflows will re-merge in a potentially inconsistent manner, I wire in an instance of an abstraction called 'Order' between the output and all its destination inputs. This instance of order is configured to explicitly control the order that the output date stream events are executed in. Then I will use a second abstraction called 'EventBlock' at the end of all data paths except one, the one that executes last.

By default multiple IDataFlows wired to a single output are executed in the order that they are wired anyway. On the diagram, they are drawn top to bottom in that order. This improves the determinism but is a little too implicit for my liking, so that is why I use the order abstraction.

As used in the coffee-maker example earlier, this paradigm simulates electronic circuits instead of using the concept of discrete messages. Semantically the inputs have the values of the outputs they are wired to at all times. This type of flow is readily implemented with shared memory variables.

FRP (Functional Reactive Programming) also is effectively a live dataflow execution model.

The use of the word synchronous here is different from its use in the discussion of synchronous/asynchronous events above. Here it means a master system clock clocks the data around the system on regular ticks. At each tick, every instance latches its own inputs and then processes them and places the results on their outputs. Data progresses through one operator per tick, so takes more time to get through the system from inputs to outputs. The result is a more deterministic and mathematically analysable system.

The execution timing and the timing of outputs occurs at a predictable tick time, albeit on a slower time scale than an asynchronous system. All timings are lifted into the normal design space.

Glitches that could occur in an asynchronous system (discussed earlier) are eliminated at the level of single clock ticks. A fast glitch could not occur. A glitch would occur when different data paths had different lengths, and would last for at least one tick duration. Controlling glitches is therefore lifted into the normal design space.

This is a thought experiment at this stage. The experiment is to see if we can do structured programming for activity flow. Remember activity flow is instances of domain abstractions, each of which generally has a start port and a done port.

The idea is to mimic imperative structured programming. Structural programming is what got rid of the goto and introduce block structured statements such as while and if. It is generally laid out with indenting that exactly matches the nested structure of braces. Your brain sees the indenting but the compiler sees the curly braces. (Except for Python which makes the compiler use what the brain sees).

In this program, we will string together some instances of domain abstractions and include a loop and a conditional. The indenting structure is the same as for the imperative version.

Remember this code is not executing the activity flow, it is just wiring it all up for later execution.

TBD need the corrsponding diagram here to show what this code is trying to do

First remember that WireIn returns its second parameter to support this fluent style. A is the first activity. A’s done port is wired to B’s start port, so B is the second activity. Everything else is in a loop. The 'Loop' function takes two parameters, one is another flow and one is the looping condition, which in this case is H. B gets wired to C. 'If' is a function that takes three parameters, a condition, which in this case is D, and two flows. C gets wired to D. The 'If' function expects D to have two done ports, called donetrue and donefalse. It wires donetrue to E. It wires donefalse to F. 'If' wires the done ports of both E and F to a null activity instance to recombine the flow. The null instance is returned by 'If'. The null instance is wired to G. G gets wired to H. The 'Loop' function expects H to have two exit ports calls done and loop. 'Loop' wires H’s loop port to C, and returns H. H is wired to I.

This code looks okay, however, as is often the problem with text based representations of relationships, most of the instances will probably need additional wiring to other things as well. If this is the case, and the requirements implicitly contains a graph structure rather than a tree structure, then a diagram wll be the best way to represent it.

A bigger problem with callback functions or futures is that if the requester call is inside structured statements such as a loop or if statement, or has been called from another function, all the code right back to main() needs to be rewritten like a state machine. It must keep state variables to remember what would normally be implicit in the program counter state, and manually store any other stack based state that the compiler would normally handle for the execution flow through the program. If the code is a simple function called directly from main, this can be done fairly easily. Each time the function is called, it reads the state, which is usually a function pointer, and dispatches to it. Callbacks or continuations go to their own functions.

+ An advantage of this style of programming is that it easily handles all time discontinuities - things that would otherwise block a thread. It allows reacting to unexpected events much more easily. And it allows longish routines to yield by simply returning part way through, say inside a loop, to reduce the latency of any other concurrent tasks waiting to execute. When the main loop calls back, it can use the state variables to resume processing where it left off.

+ The great disadvantage of this style of programming is when the program is more prescriptive than reactive. There is a fixed sequence of things that will happen, and we want to express that as normal sequential lines of code, even though certain operations will block. For example, we are moving a large amount of data. Exceptions to the prescribed sequence are rare. I find that async/await or co-routines are the best solutions for this situation. If they are not available, then a cooperative (non-preemptive) thread could be considered to solve the one situation.

+ One of the most common requirements for concurrency is responding to user input. For this we may specify a soft deadline of 0.1s. This means that all state machine, callbacks, or other run-to-completion routines should execute in less than 0.1s. This not difficult to do because the vast majority will execute very quickly. What I sometimes do is put in a system timer to measure the longest running routine. It’s usually updating a large display.

+ What I see happening in most traditional systems is that once an RTOS is included in the system, it is considered to be the solution to all concurrency in the system. But probably 99% of concurrency in most systems can be done on a single thread. Most tasks may have priorities, but will wait until the CPU resource gets to them. So what I do is avoid using threads except for when the specific case of performance can’t be solved in any other way. So, in my entire career in embedded systems, I have never ended up having to use a second thread, even when I have an RTOS already in the system at my disposal. Short interrupt routines have handled all situations with hard real time latency requirements. The state machine programming style has better suited the reactive nature of most embedded systems.

+ Remember you can only have one highest priority thread. If you are really in a situation where you have one or more hard real time deadlines that can’t be done in interrupts, then you should probably consider putting in multiple MCUs rather than trying to do, for example, rate-monotonic analysis.

+ Of course, if your system has multiple CPU cores, then you probably have a performance requirement that will need multiple threads to make use of them.

TBD

Now let’s switch our attention back to another example of maintenance. Let’s add the tiebreak feature. Another instance of Frame will score the tiebreak quite nicely. However we will need an abstraction that can switch us from playing the set to the tie break. Let’s call it Switch, and give it a lambda function to configure it with when to switch from one subframe tree to another. Switch simply returns the sum of scores of its two subtrees. Here then is the full description of the rules of tennis:

And here is the code version of that diagram. This application passes an exhaustive set of tests for the scoring of tennis.

And just like that we have a full featured executable tennis scoring engine.

Notice that I have added string names to the instances of Frame and other objects. This is not required to make the program function, but generally is a good habit to get into in ALA. It is because in ALA we typically use multiple instances of abstractions in different parts of the program. The names give us a way of identifying the different instances during any debugging. Using them I can Console.Writeline debugging information along with the object’s name.

Around 8 lines of code express the rules of ten-pin bowling and around 15 lines of code express the rules of tennis. That sounds about right for the inherent complexity of the two games. The two rule descriptions actually execute and pass a large battery of tests.

The domain abstractions are zero-coupled with one another, and are each straightforward to write by just implementing the methods of the IConsistOf interface according to what the abstraction does. The abstractions are simple and stable. So no part of the program is more complex than its own local part.

The domain abstractions are reusable in the domain of game scoring. And, my experience was that as the details inside the abstractions were implemented, the application design didn’t have to change.

Why two example applications? The reason for doing two applications in this example is two-fold.

Here is a link to the code on Github: GameScoring code

When a new project begins, the only new information we have is the requirements. Any design decisions that don’t depend on the specific requirements could already have been made beforehand. It is those decisions that form the ALA reference architecture. Therefore, when we get the requirements, that is our immediate and total focus. We may not know all of them, but we will only need a sample to build start inventing our domain abstractions.

Looking at the available new information as a whole first instead of taking it a bit at a time during the project’s sprints will make a huge difference to the quality of the abstractions we invent, and eventually the quality of the application’s architecture.

The process in the first iteration takes requirements one by one, and represents them, in all their detail. Domain abstractions will be invented as you go, and they will have parameters or properties that will handle those details from requirements.

For the first green field application, you spend a maximum of one sprint. After that you do need to find out if your design works. So you may not get through all the known requirements. That does not matter.

To know whether knowledge from your design goes in the application layer or the domain abstractions layers, you consider what the scope of that knowledge is. Is the knowledge specific to this one requirement in the one application, or is it potentially reusable in the same or other applications? A softkey label is clearly specific and goes in the application. The concept of softkeys is clearly a domain abstraction.

The output of the first sprint does not implement any of the invented abstractions, but it does include all details of the requirements that are looked at. In so doing, you design the first approximation of a DSL. The DSL may be refined later as more requirements are looked at.

Each abstraction will eventually be implemented as a class, but initially we just record the names of the abstractions, and a short explanation that provides the insight into what the abstraction does.

By the end of the first sprint the requirements will have become easier and easier to represent, as the set of abstractions will have taken shape. Sometimes you will generalise an abstraction further to enable it to be useful in more parts of the application.

By keeping moving through the requirements at a much faster pace than in normal development (say one feature per hour instead of one per week), we can keep representing them in a coherent way, revising abstraction ideas we have already invented. Ideally, we will end up with a set of domain abstractions that can be wired together in different ways to represent the requirement space - the space of all possible requirements. That space, which we call the domain, will grow slightly as time goes on and it accommodates a growing family of products, but we don’t want it to grow beyond that. We don’t want to invent ways of implementing things we will never do.

On the other hand, we do want to invent quite a few abstractions during this first iterations so that we end up with a coherent set of them that will compose together in an infinite variety of ways.

The output of sprint zero is usually a diagram showing the wiring of instances of abstractions, together with configuration information for those instances.

It doesn’t matter if some of the requirements are wrong. Chances are they are still useful for scoping out the domain space. What we are actually producing in this phase are the necessary abstractions for the Domain Abstractions layer. If the requirements change later, it will be trivially simple to change them as only the Application layer wiring should change.

Once this process has started to become easy, which should happen within the first sprint, the burning question in our minds will become “Will all this actually work?” We have to trust that there will be a way to write those abstractions to make the whole system execute.

The most difficult part for people new to ALA is the skill of inventing the domain abstractions. You can fully understand the theory of ALA and why we need to invent abstractions, and still find it tricky to actually invent them. That’s because in this step in the process, we have to literally become inventors who are adding to the state of the art within your domain.

So the biggest insight I can offer here is don’t expect it to come without effort. Put your inventors hat on. Deliberate about the problem over a long period of time. Leave it overnight. Come back to it afresh in a week. Be patient.

I sometimes find that a solution will pop out of nowhere while reading about something completely unrelated. Once the human mind has been deliberating for some time, it remembers the whole context of the problem, and seems to carry on pattern matching for potential solutions in the background. I am reminded of the story of the Dam Busters. The British wanted to bomb the German dams to flood the industrial valleys below them. They worked out that they could deliver the bombs to the face of dam by skipping them across the surface of the lake. But they couldn’t get their bomber low enough to the water at night to get them to skip, because they had no way of knowing their height above the water.

Then one of the pilots was attending a play. He saw the spotlights pointing at the actor from different angles. And suddenly he had the solution. By mounting spotlights at the nose and the tail of the aircraft at a certain angle, they would know when they were at the correct height when the two spots came together at the surface of the water.

In my experience, there is always a solution, and it is always worth the effort. Just as good inventions are like gold, good abstractions are like gold in your application.

This section offers some practical tips on where to start.

Your classes and functions are now proper abstractions because they don’t know anything about the outside world, including who they are wired to, making them reusable.

In the second sprint we start converging on normal Agile. You pick one feature to implement first. Agile would say it should be the most essential feature to the MVP (minimum viable product), but this can be tempered by the need to choose one that requires a fewer number of abstractions to be implemented. Next, you design the interfaces that will allow those abstractions to plug together according to the wiring you already have from the first sprint. What messages will these paradigm interfaces need to pass at runtime between the unknown clients to make them work?

It may take several sprints to produce the first working feature, depending on the number of abstractions it uses.

At first this sounds as if it might be just the waterfall method reincarnated. Do an overall design, document it or model it, and then write lots of code before everything suddenly starts working. But the design we created in iteration zero is very different from what a normal waterfall would produce, and is resilient to the sorts of problems waterfall creates. Instead of a ‘high level’ design of how the implementation will work which is lacking in detail, the design is a representation of requirements, in full detail. The design is not a model. It is executable.

There is one more important thing that the design phase in Iteration Zero does. While it deliberately doesn’t address any implementation, it does turn the remainder of the implementation into abstractions, and those abstractions are zero coupled. To convert from executable to actually executing, it only remains to implement these now completely separate parts. You can give these abstractions to any developer to write. Together the developers will also easily be able to figure out the paradigm interface methods needed to make them work, and the execution models to take care of the execution flow through them with adequate performance.

Often when a project is split into two phases, the first phase turns out to be waste. The devil is in the details so to speak. This happens because the implementation details in phase two are coupled back to and affect the design in phase one. As learnings take place during implementation, the design must change. In ALA the output from phase one is primarily abstractions, which are inherently stable and therefore hide details that can’t affect the overall design. If the abstractions are good, phase two will typically have little effect on the work done in phase one.

Once the first feature is working, several abstractions will have been implemented. The second feature will take less time because some of the abstractions are already done. In ALA velocity increases as time goes on and keeps increasing until new features only involve instantiating, configuring and wiring domain abstractions in new ways. This velocity acceleration is the complete opposite of what happens in monolithic code.

Imagine going into a sprint planning meeting with a Product Owner, a small team of developers, and a mature ALA domain that already has all the common domain abstractions done. As the Product Owner explains the requirements, one of the team members writes them down directly as they would be represented in terms of the domain abstractions. Another team member watches and remembers any lost details without slowing the product owner down. A third member implements the acceptance tests in similar fashion, and a fourth provides him with test data. It would be nice to have a tool that compiles the diagram into the equivalent wiring code. With such a tool, the team could have it executing by the end of the meeting. At the end of the planning meeting the development team say to the product owner "Is this what you had in mind?". The team can get immediate feedback from the Product Owner that the requirements have been interpreted correctly.

Of course, the planning meeting itself would only produce 'normal' functionality. Usually it is up to the development team, not the Product Owner, to uncover all the abnormal scenarios that can happen, and that is usually where most of the work in a software system goes. Having said that, in a mature domain, the validation of data already has decorator abstractions ready to go.

The graph shows the effort per user story against months into a green-field project. The left axis is arbitrary - the shape of the curves is what is important. For a big ball of mud, experience tells us that the effort increases dramatically and can asymptote at around 2 years as our brains can no longer handle the complexity, and the project must be abandoned.

The COCOMO model, which is an average of industry experience, has a power relationship with program size, with an exponent of around 1.05 to 1.2. I have used the mid point, 1.1, for this graph. The model appears to imply that getting lower than 1.0 is a barrier, but there is no reason to believe this is the case. Reuse can make the power become less than 1. The range of 1.05 and 1.2 probably results from some reuse mitigating some ever increasing complexity.

ALA takes advantage of the fact that zero-coupled abstractions can keep complexity relatively constant and drastically increase reuse. A spectacular fall in effort per user story is thus possible.

After they have modified the diagram, the maintaining developers will need to manually modify the corresponding code. Here they will see instances of abstractions being used all over the place, either 'new' keywords or function calls. If we have done our job with knowledge prerequisites, they will have been introduced to these abstractions. However, it doesn’t hurt to have brief reminders of what they are pop up when the mouse is hovered over them. So put in triple slash comments (or equivalent) describe the abstraction succinctly, with the intention of it being a reminder to someone who has already met the abstraction. Put a full explanation in the remarks and examples sections.

The class name after a new keyword is actually the constructor name, so you must duplicate the summary section there. Often in ALA, the class name is not referred to at all in the application.

First notice how the entire calculator is there. Every detail for this particular calculator is represented in the diagram. All those details cohesively describe the calculator. This is an important aspect of ALA designs. All the UI and all the dataflows to make a working calculator are in this one diagram. What is not here is the details we left to the domain abstractions. None of these abstractions is specific to a calculator. They can be used for all sorts of things. Even the Formula abstraction would be useful in any application where a calculation needs to be changed at run-time. For example an insurance application may need configurable calculations.

The left side shows two instances of the abstraction, Horizontal. These arrange their children horizontally in the UI. To the left of those (not shown) is an instance of Vertical, which arranges the two Horizontals vertically. And to the left of that, also not shown, is an instance of MainWindow.

Each Horizontal has 3 instances of TextBox and one Text. The TextBoxs allow you to enter a string. The abstraction, Text, can only display a string. I see at this point, I hadn’t put in the TextBox for the description.

We can ignore the use of labels for a moment and just enter a formula containing constants into the first row TextBox. Let’s say we type in "2+1". The string "2+1" goes along the dataflow wire to the StringFormat instance on its port 0. The StringFormat is configured with the format string "({1}⇒{0})". StringFormat uses this format string in the same way as an interpolated string in C#. The {0} is substituted with the string coming in on port 0. The {1} is substituted with the string coming in on port 1. Since we have no string on port 1, the output from the StringFormat will be "()⇒2+1". This is simply a lambda expression with no parameters. This string is fed into the instance of Formula. Formula is an abstraction that knows how to evaluate a lambda expression. Actually it will accept just a formula string (such as "2+1") as well. We can ignore the other input of Formula for the moment. Formula will evaluate "()⇒2+1" and produce the number 3 on its output. This output is a dataflow of type double. This number is fed to an instance of NumberToString, and from there it goes to the instance of Text that knows how to display a string.

Now let’s follow the use of labels in the calculator. Let’s put the labels "a" and "b" into the TextBoxs for labels on the two rows. "a" and "b" are fed to the Concat instance. Concat’s input port can have any number of string dataflows wired into it. In this diagram it has only two. What Concat does is concatenate all its inputs adding a separator. In this case the separator is configured to be a comma. The output of the Concat is "a,b". The concatenated list of labels is fed into port 1 of both StringFormat instances.

Now let’s put the formula "a*3" into the 2nd row of the calculator. The output of the StringFormat for that row will be "(a,b)⇒a*3". That lambda expression will be fed to the Formula instance, which will evaluate it, using the value on its first input port for the value of 'a'. The output will appear on the corresponding Text in the 2nd row.

So that’s all there is to understanding how the calculator works. At this point it takes a leap of faith that the abstractions can all be made to work, and that the two programming paradigms used, the UI layout and the dataflow, can be made to work. Not withstanding that, all the information required in the design of the calculator is captured.

At this point I drew little drawings of all the invented abstractions. Actually I reused TextBox, Text, Vertical, Horizontal and FormatString from a previous project. And I had already made the UI and dataflow programming paradigm interfaces in previous projects, so I reused them as well.

I see I forgot to draw Concat. I had to write that one. Here is its template as it was in my Xmind templates diagram.

The two ports are both drawn on the right hand side unfortunately - a limitation of using Xmind as the drawing tool. Drawing the templates makes the abstractions clearer for their implementation.

Abstractions are generally trivial to implement because they are zero coupled with anything. They are like tiny stand-alone programs. Here is the full code for StringConcat.

The code in Concat is straightforward if you know C#, except for a few conventions which are to do with the use of ALA (which I was already proficient with):

For reference, here is the IDataFlow interface, which lives in the ProgrammingParadigms folder, and is used by most of the domain abstractions including the StringConcat abstraction:

As you can see, this interface is simple, but its importance in giving us a dataflow programming paradigm is huge. It allows objects to be wired together so that data can flow from object to object without the abstractions knowing anything about each other. Note that IDataflow uses a 'push' execution model. This means that the source always initiates the transfer of data on its output (by using 'set'). The IDataFlowB interface is a little more complicated, so we wont go into how it works just now. But it achieves exactly the same purpose of the dataflow programming paradigm, just in a different way that allows more than one input port of the same type. .

The other domain abstraction I needed to write for the first time was Formula. Here is the template as used in Xmind:

Once again, one of the input ports is shown on the right when we would prefer it to be on left.

Formula can be configured with an optional C# lambda expression when it is instantiated by the application, for example:

If used in this way, the formula is fixed at design-time. But its real power comes from the fact that it can take a formula as a string at run-time. Formula has an input dataflow port called "formula".

The Formula abstraction has to parse the formula string and then evaluate it. I used Roslyn to do this in a few lines of code. These lines of code took me a few hours to figure out however:

As you can see, currently the Formula abstraction can only handle formulas that use a maximum of six parameters. The calculator application can use any number - it’s just that any one formula is limited to using only six.

The rest of the code in the Formula abstraction is mostly dealing with this requirement of exactly six parameters used by LambdaType. But that’s all internal to the abstraction. An instance of the Formula abstraction can handle any number of operands from zero to many. The Formula abstraction is also tolerant of the string on the formula input port being either just a formula such as "2*(3+1)" (implying it’s not using any operand inputs), or a proper lambda expression such as "(x,y,z)⇒x*(y+z).

Now that we have our needed domain abstractions, let’s return to the application layer, and see if we can get this calculator running.

First here is the startup code for a wired ALA application. It is very simple:

Here is an ALA Hello World application

Ok, now we are ready to hand wire the hand drawn calculator diagram shown above:

Commentary on this wiring code

Handwritten wiring code like the above can be managed for small applications. It is easy to get it wrong though. The code is not readable from the point of view of understanding the application. That’s what the diagram is for. So every time you make a change to the requirements, you need to do it on the diagram first, then update the hand-generated code. You are still better off doing this than not using ALA. If not using ALA, the relationships of the cohesive, explicit ALA diagram still exist, but they become obscurred and distributed inside your modules.

But we don’t have to hand generate code. At the time of writing a hand written graphical IDE is under development.

Here is the calculator row part of the diagram, which was successfully used to auto-generate code:

Before we had that tool, we used Xmind to do diagrams from which we could generated code. This is the approach described in the next section.

Here is the diagram of the 2-row calculator as it was originally entered into Xmind. You can click on the image to get a bigger version.

Xmind is not the perfect tool to do this, but it has one huge advantage - it lays itself out. This is so important that it’s why we use it (until our new tool is ready). We will put up with the disadvantages, and the necessary conventions until then. Those conventions are documented in the wiki section of the project in Github here: https://github.com/johnspray74/ReactiveCalculator/wiki

Entering the hand-drawn version of the diagram is a simple matter of copying and pasting the Xmind templates for the abstractions to the right place in the diagram. This connects most of the needed wiring from port to port. Xmind supports tree structured diagrams, so any cross tree wiring was done by using the red lines, which are quick to put in with a shortcut key.

The Xmind version of the diagram is pretty much identical to the hand drawn version. The colored boxes are instances of abstractions. All the other nodes attached around those colored boxes are the ports. The < and > signs in the ports are significant, and tell the automatic code generator which way to wire the instances. The asterisks are also significant, and tell the code generator that many wires can be wired to the one port.

The diagram done, it’s time to generate the code. A tool called XmindParser does this. It can be downloaded from the Github project main page. Here is the tool in use to generate a calculator from the diagram.

The tool can put the generated code into your Application.cs file if you give it special markers like the one below.

The markers contain the name of the Xmind diagram they get code from. This allows several diagrams to be used for one application.

I usually put the markers inside a function so that all the instantiated objects are private to the function.

Here is the code again with three lines of generated code shown in each section.

Completing the diagram had taken another morning of work. But the calculator was now working and I was ready for the workshop.

I made a diagram with six rows, but it was getting pretty large, and the duplication was pretty clumsy.

The 6-row calculator is powerful compared to any normal calculator, but still wouldn’t do jobs like the one shown here which has 11 rows:

So it was time to do some maintenance, and solve that problem of the repetition in the diagram at the same time. During this maintenance, none of the existing abstractions changed. Their ports stayed the same. Their internals were improved a little in some cases but nothing significant. Formula was modified internally so that it could handle more than six operand inputs, although any one formula can still only use six of them.

There were two major changes though. One was to put the repeated wiring for a calculator row inside its own abstraction. This abstraction is called "CalculatorRow". This abstraction is less abstract than the Domain abstractions it uses, but more abstract than the application, which will use it multiple times. Other version of the calculator alos reused it. So it goes into its own layer between the two. This new layer is called "Requirements Abstractions". We don’t make new layers lightly, but we had had the experience in a larger project that the diagram got too large. We needed to factor out some of it as 'Feature level abstractions'. These abstractions needed a new layer between the Application layer and the Domain abstractions layer. So I was reasonably happy to make use of this new layer in the calculator to reduce a now quite large diagram.

Here is the Xmind template for the CalculatorRow abstraction showing its ports:

The implementation of CalculatorRow was done with a diagram using the new Graphical ALA tool which was shown earlier.

When you implement an abstraction by an internal diagram, there needs to be some extra code to wire from the ports (shown in the template above) to the internal wiring. I found this code quite tricky the first time and it took me a while. But there is a pattern to it. The new tool will be able to automatically generate this code as well, but for now I did it by hand.

I tested the completed CalculatorRow abstraction by making an application that uses it twice:

Because this test calculator has only two rows we can directly compare it with the 2-row calculator above and see how the two rows are now represented by two instances of this new CalculatorRow abstraction. However we are still not there to building a calculator with 10 rows. If we were to instantiate CalculatorRow 10 times, it would need 100 wirings in Xmind to connect all ten results to every CalculatorRow’s operands input. Having thought the 36 wirings of the 6 row calculator was nuts, there was no way I was going to do 100 of them by hand.

So the next step was to invent a domain abstraction called 'Multiple'.

What 'Multiple' does is you instantiate it in your application and configure it with the number you want. In this case N:10. Multiple has a port called factory which uses a new interface called IFactory. Then what you do is add a small Factory class inside any abstraction that you want multiple instances of. In this case we want multiple instances of CalculatorRow, so I added a small class to that abstraction called CalculatorRowFactory. CalculatorRowFactory implements IFactory (which resides in the Programming Paradigms layer). Multiple can now create many instances of the abstraction that is wired to it. But these instances now have to be wired into the rest of the wiring in the application diagram as needed. To accomplish that, multiple is configured with two methods. These methods, which are part of the application, know how to wire the new instances into the rest of the wiring.

Here is the Xmind template for Multiple:

Using the new Multiple abstraction, we can now build a 10-row calculator. Here is the new diagram:

You can see inside the instance of Multiple that it is configured to make 10 of whatever is attached to its factory port. The CalculatorRow abstraction is shown next to the CalculatorRowFactory so you can see what it makes. You can see inside Multiple the two methods that Multiple calls when it makes a new instance. These methods are used to wire the CalculatorRow into the rest of the application wiring.

Getting to this point had taken another two Saturday mornings of work. It was mainly spent on thinking out the patterns for how the internal wiring inside CalculatorRow should get wired to CalculatorRows own border ports. Also in the initial attempt I had allowed temporal coupling to creep in between abstractions. It mattered whether the application wiring was done first. And if you let coupling creep in bugs will happen. The temporal coupling was resolved by making CalculatorRow not care whether or not the external wiring is done when it is instantiated.

Wouldn’t it be cool if the calculator started with say 4 rows, and had a button for adding additional rows? You will notice that the Multiple abstraction template above has a port called addRow. It takes an IEvent. If it receives an event on that port at run-time, it will create another row and call the lambda wiring functions to get that row wired in.

Here is the calculator with the button added to the UI. This is the calculator from which the screen shot shown above was taken.

I wanted the Button to be at the bottom, so I added another instance of Vertical to act as the wiring point for new row to be attached to the UI. The output of the Button is another programming paradigm interface called an IEvent.

Here is the source code of IEvent:

Again, considering that this interface enables a whole programming paradigm, it is extremely simple and abstract, consisting of only a synchronous function call. It allows any objects that can produce events to be wired to any object that can receive an event. In the CalculatorNRows example, it allows a Button to be Wired to a Multiple. When the Button is pressed, Multiple creates a new row.

There is one major Programming Paradigm that we have not explained yet. It is the one that allows UI instances to be wired together. The meaning of wiring two UI objects together is that one contains the other. Here is the IUI interface that implements this programming paradigm.

The interface simply allows a containing UI instance to get the WPF (Windows Presentation Foundation) element from the contained UI instance it is wired to.

Making Multiple able to add rows to the calculator at runtime took yet another Saturday morning. But it was worth it to solve the challenge of learning how to change the wiring at run-time. The wiring diagram now statically describes how it dynamically changes itself.

That concludes the story of the development of the calculator using ALA. The full source code, and all the diagrams for every version along the way are in the Github project at https://github.com/johnspray74/ReactiveCalculator

In the example so far, we assumed, based on the imperative code, that the Compose function directly and immediately calls the functions. When we compose the functions as a dataflow in the top layer, we assume that under the covers in the Compose function, the execution flow will be the same as the dataflow.

However, it is quite possible for the execution flow implemented in the compose function to work entirely differently from the dataflow. We can, for example, implement deferred execution, where the Compose function builds an executable structure which can be run later. This separation of how it executes from the declarative composition of the functions as a dataflow is our forth advantage of using a Compose function. It allows us to build Compose functions that use deferred execution.

The Compose function we had in the previous section evaluated the functions immediately and returned a result directly. Let’s write a deferred (or lazy) version of Compose. Instead of returning the actual result, this version will return a new function that represents the composed functions. This returned function can then be called later. Here is a deferred version of the Compose function:

It’s the same as our previous Compose function except for the type of the first parameter and the return type. In the immediate execution version, the type of the first parameter to compose was an int, which comes from the execution of the previous function in the chain. And the return value was also an int. Now these two things are functions because we are not evaluating the functions as we go, we are composing the functions to return a new single function.

The () ⇒ syntax is a lambda expression for a function that takes no parameters. It creates a new function that calls the source function and then calls f.

What the Compose function returns is actually an object structure made up of delegates and closure objects created by the compiler:

The purple boxes are C# delegates. Delegates can be thought of as a pointer to a method. Because the method is in an object, a delegate is actually a pointer to an object together with a reference to the method in that object’s class. Delegates themselves are objects. The delegate object has a pointer to another object (called Target) plus a reference to a method in that object (called MethodInfo).

The yellow box is a closure object. A closure is an object made from a compiler generated class with a single method. It can have fields which are references to variables in the local scope. In this case, the closure object has two fields, which are both delegates. The closure’s method calls the first delegate, and then calls the second delegate, passing the result returned by the first to the second. The structure is returned as a delegate that points to the method in the closure object.

The structure looks surprizingly complicated consisdering the code that generated it was just '() ⇒ f(source());' That doesn’t matter because its all generated by the compilier. I wanted to show it so we can see all the code for this deferred version of the compose function.

Now that we have a deferred version of our Compose function, we can use it like this:

That’s just the same as our previous top layer code, except that we show how we can get a function back which we can save to be executed later. It’s like we got a small program back.

When we want to execute the combined function, we can do it like this.

We will usually prefer to build deferred versions of Compose functions, just because that gives us the versatility to execute them now or later, or even to execute them many times. In other words, Compose can build a program.

Note that the deferred version of Compose takes a function as its first parameter, not a number like 42. So we couldn’t pass 42 to it to start the chain. Instead we used another extension method called ToFunc to get a function that returns 42.

Deferred function composition generally returns a surprisingly large object structure containing delegate objects and closure objects all generated by the compiler. This is one reason why monads are so hard to understand. Here is what the object structure for the composedFunction above looks like:

You can click on the diagram to see it enlarged. On the left side of the diagram you can see the four closure objects that implement the 42 and the three lambda expressions. Each of these closure objects has a delegate object that is used to reference it. Then there are three other closure objects that were created by the Compose function. These closures call the other closures via their delegates in the correct order. The entire structure is returned as a single delegate on the right hand side.

There is a fifth advantage to using a Compose function. This advantage is big, and is what allows us to finally call it a monad.

In imperative code we might typically need some extra code after every function call. A common example would be to check for errors returned by one function before calling the next function. If we have a compose function, we can put that extra code inside the Compose function instead. This refactoring is essentially what the monad pattern is.

We’ll give four examples of imperative code that needs some extra common logic after every function call. In each case, the functions we are composing are not returning a simple value that can be fed directly to the next function. They are returning something else, so in every case we need a little bit of extra logic to handle what the function returns before calling the next function.

It’s what this extra logic code does that distinguishes one monad from another.

Composition of functions that can fail.

In this first example, we may need to allow for the fact that functions can throw an exception, or return null, or a Maybe object or even -1. For example, the function may be vulnerable to a divide by zero. In imperative code, we would commonly have to add if statements or try statements so that we don’t call the rest of the functions in the chain when something goes wrong. That will likely create nesting, arrrgh.

In C code, returning -1 is often used for this purpose, so let’s use that for our first example just because it’s so simple. Here is the imperative code:

Note that, even though we are using intermediate variables, we got our nasty nesting back. Let’s see how we do that the monad way by refactoring the if statements into the Compose function:

Composition of functions that return many values.

We may have functions that return many values, such as an array, a list, an IEnumerable or an IObservable. We then want to feed all the individual values into the next function, which will in turn return multiple values, and then recombine all the values nto a single array or list. In imperative code, we do this with nested for statements. For example, the function may be given customers one at a time and returns a list of their orders, which we want to join back into a single list of orders.

Again note the nested levels of brackets and indenting for every foreach.

Composition of asynchronous functions.

The functions that we want to compose may contain delays, or they may wait for input or output, or they may wait for processing occurring on a different thread or processor. In other words, the functions may be asynchronous - they will return a result later, not immediately.

Because these type of function don’t return a value immediately, we can’t write an immediate execution monad. But we can still write a defered execution monad - a BInd function that will combine asynchronous functions, and return an asynchronous function.

There are different estanblished ways to implement asynchronous functions. The oldest is the callback. The function receives a callback function as an extra parameter. When the result is ready later, the callback function will be called to pass back the result.

More recently, asynchronous functions are implemented by returning a Task or future object. For our purposes here, a Task or future object are the same thing. They are an object that will have a result placed into it at a later time.

Here is the imperative version of the application layer code that composes functions that use callbacks. The common logic between the functions, function1, function2, and WriteLine is to create a callback function to be passed to the next, which is done using lambda functions:

Here is the imperative version of the application layer code that composes functions that return future objects. The common logic between the functions is to put a continuation function into the future object that was returned by the previous function:

In either case, notice the nasty indenting for every function we want to compose. In this case we could eliminate the indenting by using Unwrap() like this:

But I want to show how monads simplify this even further.

(Note that both versions require lambda expressions (for example, the lambda expression starting with 'task1 ⇒" ). In the first implementation above, the lambda expression is an Action. In the second implementation the lambda expression is a function. So they are two different overloads of ContinueWith. In the second implementation, the lambda function returns the type returned by the function, which is a Task<T>. So ContinueWith returns Task<Task<T>>. The Unwrap discards the outer Task.)

Of course, async/await also simplifies this particular example, but I want to show how monads can do it first.

Let’s compose functions that return angles.

The three examples of monads that we did so far, the MinusOne monad, the List monad, and the Task monad are common monads in the industry. (Well, not the MinusOne monad, I made that one up, but the industry version of that is the Maybe monad, which we shall switch to when we look at deferred versions.)

For our last example, I wanted to do something custom, just to show that you really can create a monad to handle any type of intermediate logic you want between the functions you are composing. As long as we are always doing the same logic between all functions we compose, we can refactor that common logic into a Bind function.

Let’s say we always want to do modulo 360 arithmetic. And let’s throw in a total rotations counter as well, which we want to pass through the chain. Here is some imperative code:

In all 4 of the above examples, the refactoring follows a pattern, which I will call the monad pattern.

In each case, we were able to create a compose function (called Bind) that just composes functions in a declarative way. In each case we were able to refactor any common logic needed in the imperative code between function calls into the Bind function. That’s pretty much what we are about with monads, composing functions using a Bind function and putting any common logic that is needed between each function into the Bind function as well. It is that common logic that distiguishes different types of monads.

The Compose function is often called Bind, but can go by other names such as the symbol ==>. If the monad is the list monad or IEnumerable monad, it can be called flatmap or SelectMany. The most common monad you will come across is the IEnumerable monad, but many other types of monads are possible.

More formerly, a monad consists of three elements:

To see how monads compare with ALA, we now want to do deferred versions of our 4 examples. Actually the Task<T> monad is already deferred, so we will just do the other three examples.

For the minus_one example, we are going to switch to the IMaybe monad first, because that’s the generally used solution to composing functions that may return no value. Skip this if you are already familiar with the IMaybe concept, unless you are interested to see how the IMaybe monad and its Bind function works.

Composition of functions that can fail by returning IMaybe<T> or Nullable<T>.

Using minus one, as we did earlier to represent a 'no value', is not used outside the C world, and limits the data itself to positive integers. The more general solution in the monad world is the IMaybe<T> monad. It’s called IMaybe because maybe it contains a value, or maybe it doesn’t.

The IMaybe version of Bind is similar to the -1 version. However the chaining interface is IMaybe<T>. Bind takes an IMaybe interface and returns an IMaybe interface, and the functions that we compose together also return an IMaybe interface.

We will have two classes that implement IMaybe. They are called Something and Nothing.

Here is example top layer application code composing functions that return IMaybe.

Application layer code

Something and Nothing are classes that implement IMaybe<T>, which we provide below for completeness.

The Bind function wont call the lambda expression if the result from the Bind is Nothing. But if the result from the previous Bind is Something, it takes the value out and gives it to the lambda function.

The Bind function takes an IMaybe as a parameter and returns an IMaybe. Notice that we need to convert the starting value, 42, to an IMaybe. That’s because the first Bind in the chain must have an IMaybe. To be a monad, we generally need to supply this method which is always used at the start of a chain of Binds.

Here is the IMaybe interface:

IMaybe consists of two getters, one called HasValue() that returns a bool to find out if a value is there, and the other called Value to get the actual value out if there is one.

You would normally use HasValue first and only if it returns true would you use Value. HasValue is analogous to the MoveNext method in the IEnumerator interface, which you also have to call first before retrieving a value. We will need two classes that implement IMaybe, one to represent a nothing, and one to represent something:

Monad layer code

The Bind function uses its input IMaybe<T> to see if there is a value present or not. If there is nothing it doesn’t call the function. It just returns a new IMaybe<U> implemented by a Nothing object. If there is a value, it gets the value and passes it to the function. Then Bind returns the IMaybe returned by the function. Here is the ToMaybe and Bind functions:

Monad layer code

All the monads we have done so far (except for the Task monad), were immediate or eager versions of the monads. This means that the Bind function calls the composed functions itself and passes the results(s) to the next function. We did the immediate versions because they are so simple.

However, we want to do deferred versions of all these monads because they are more versatile, and they create an object structure that is returned for later execution. We want to compare this object structure with the way ALA also creates an object structure composed of domain abstractions.

With deferred monads, we can do either pull versions or push versions.

For the pull version, we keep a reference to the last object in the structure. We call a method in that object when we want the result. That call pulls the data through the chain of objects. For the push version, we will keep a reference to the first object in the chain. When we want a result we will tell the first object to start, and the result will pop out of the other end.

In ALA, we generally default to programming paradigms that use pushing. To compare monads with ALA, we will therefore want to understand the push variations of the monads. However, pull monads are more common in the monad world. So we will do both. It gets pretty interesting to see the differences between the two in terms of how everything works in the code.

The implemtation code for each of the deferred versions will obviously be a little more complicated than the immediate versions of these monads. It will involve using objects to build a structure that can be returned for deferred execution. However, for completeness, I have not shied away from including all the code for the three monad examples. I have done this for both pull and push versions. It’s not necessary to read or understand all the code. But when you want to know exactly how something works, at least the code is there along with notes to explain it. I also include object diagrams explaining the object structure that is created beneath the covers by the Bind code.

Once you have written top layer application code for composing functions using a deferred monad, you may be wondering how you would execute the returned structure of objects to get he actual value. Well, given an object, s, that was returned by the monad expression, here are examples of the ways of forcing it to execute for various types of common monads:

We will write a deferred version of Bind that composes functions that can fail by returning IMaybe<T> or Nullable<T>. It is deferred, so it returns an object that implements IMaybe in a way that will evaluate the result when the Value is required.

Here is top layer code to use the deferred/pull implementation of the maybe monad.

It looks the same as the immediate version. But it returns an IMaybe that’s implemented by a large object structure instead of returning one of the two concrete IMaybe value objects.

First we define the IMaybe interface, which is the same as for the immediate version above. The MaybeNothing and MaybeSomething classes are also the same as before. Here they are again.

The Bind function is different as it must build a structure that can be run later.

As you can see, Bind simply instantiates new class called "Maybe" that implements IMaybe, which will do all the work at runtime.

The code that runs later in the Maybe class is the HasValue and Value getters. They do all the work.

This diagram shows the resulting structure from our little bit of application code above:

Because this is a pull implementation of the monad, the references go in the opposite direction of the dataflow - from destination to source or from right to left. When you want to run the combined function, you pull the value from the right end.

Having done the deferred pull version, we will now do the deferred push version, which has an interesting analogy with the IObservable.

Here is the top layer code, which in this case returns a IMaybeObservable.

I’ve purposely left the lambda expressions out for now. Well get back to them in a minute.

For the push version we need two interfaces. One, which we will call IMaybeObservable<T>, is the chaining interface that Bind takes and returns. The other, which we will call IMaybeObserver is for doing the actual pushing of data at runtime. The two interfaces work in opposite directions. That is IMaybeObservable<T> is always implemented by the previous object in the chain . IMaybeObserver is always implemented by the next object in the chain.

Here are the two interfaces:

That IMaybeObservable<T> implementation may at first seem a little strange. It has no way of getting the value out. That’s because we are implementing a push style monad. The IMaybeObservable<T> interface is used by Bind only to get a reference to the previous object in the chain. It then gives the previous object a reference to it’s next object, which the previous object will use to push the data. This second reference will use an interface called IMaybeObserver. And yes, these two interfaces are exactly analogous to the IObservable and IObserver interfaces in reactive extensions.

IMaybeObservable<T> is used by the next object in the chain.

The IMaybeObserver interface is used by the pervious object. (We could have chosen to use C# style events (observer pattern) instead of using IMaybeObserver, but it’s easier to do it exactly the same way as IObservable and IObserver).

Note that the IMaybeObserver interface’s methods are actions. They don’t pull a value like in the pull version we did of this monad, they push. I am tempted to rename them PushValue and PushNoValue to make this obvious everywhere that Value and NoValue are used.

Normally with monads, the type that the composable functions return would be the same as the chaining interface. So all functions that are composable by this monad should return IMaybeObservable, like this type:

Having all your functions return an IMaybeObservable would certainly work, but the functions would be a little complicated. They would all have to create an object that implements the IMaybeObservable interface to return. It would be far simpler if the functions were passed an IMaybeObserver instead of returning an IMaybeIObservable. So they would have this form:

Now when the functions run, they don’t need to create an object, they just directly push the result out via the IMaybeObserver<U>> interface that was passed to them. So that’s what we will do in our example. It’s a more loose interpretation of the definition of monad, but it’s just more sensible to do it that way.

Note that the IMaybeObserver interface (listed above) could have been written with a single method like this:

However, to make it as easy as possible for the composable functions to use the interface (not have to create a Something or Nothing object), I have changed the interface to be two methods:

So now we can write the application layer code using simple lambda expression syntax:

Note that we keep a reference to the source of the chain, not the end. That is the object returned by ToMaybe(). This object has a Run method. The structure starts executing when the Run method is called. This is a departure from the way IObservable and IObserver work. With IObservable, the Subscribe method both wires the IObserver in the opposite direction, and tells the source to start. So even though it’s suppossedly a push style, execution is started from the destination end, which makes it look like a pull style. This loses some of the advantages of using a push style, for example, when it is the source that wants to initiate a push whenever the source changes, or when using asynchronous communication across a network. I really think the straightforward push system is conceptually purer and more useful. So that is what I have implemented.

Here are the ToMaybe and Bind functions for the IMaybe deferred push monad:

Here is an object diagram of the complete expression.

You can see that although using the IMaybe monad Bind function from the top layer to compose three functions is extremely simple, the structure of objects that is generated under the covers is relatively complicated. It’s no wonder that these monad things seem so hard to understand at first.

The references between the objects, which use IMaybeObserver, go in the same direction as the dataflow. IMaybeObservable is only used for wiring the structure up.

So far we have done deferred pull and deferred push implementations of the IMaybe monad. Now lets do the deferred verson of the List monad, the IEnumerable monad

Composition of functions that return many values, in this case an IEnumerable.

The IEnumerable monad is the deferred version of the list monad we did earlier. The IEnumerable monad is the most commonly used monad, and is what LINQ is based on.

The Bind function for the IEnumerable monad is called SelectMany in C#. SelectMany is not used as often as Select. Select takes a simpler function that returns U instead of IEnumerable<U>, so it doesn’t expand the number of items, it just does a one-to-one mapping. While Select is used more often, it is the SelectMany function that makes it a Monad. Here in our example application we will use three SelectManys in a row. Each will expand in number by 3, so we will end up with an IEnumerable with 27 items in the end.

Here is example top layer code that composes functions that return IEnumerable

Remember that for the IEnumerable monad, function1, function2, and function3 take a single value and return many values in the form of an IEnumerable.

In the immediate example above that returned lists, the lambda expressions looked like this:

While this will run fine when using the IEnumerable version of Bind, it’s not really in the style of a deferred monad to create memory hungry arrays. So let’s write functions that will do the same job in a deferred way:

The yield return keyword causes the compiler to generate an IEnumerable object, which it returns. The IEnumerable object contains a state machine where each state executes code till it hits the next yield return statement.

Let’s just reuse that function three times in our composed function:

The Bind function (SelectMany) for this type of monad takes an IEnumerable<T> and returns an IEnumerable<U>. The Bind function doesn’t use a for loop immediately as that would defeat the laziness. Instead the bind function uses an object that keeps state. Let’s call this object the output IEnumerable. The output IEnumerable knows how to use the source IEnumerable<T> to get the first value, which it gives to the function. The function returns an IEnumerable<U> which we will call the function return IEnumerable. The output IEnumerable then knows how to get the values from the function return IEnumerable<U> and return them one at a time. When it has exhausted all of them, the output IEnumerable<U> then gets the next value from the source IEnumerable<T>, and gives that to the function. The function again returns an IEnumerable<U>. This process continues until the source and function output IEnumerables are both exhausted.

In C#, the Bind function is really easy to write because the compiler can build an IEnumerable for you using the yield return syntax:

Note that the code in the function does not run when this Bind function runs. The compiler sees the yield return and builds an object containing a state machine that implements IEnumerable<U>, and returns that.

Since our purpose is to show how the Bind function is a refactoring of imperative code, here is a version that doesn’t cheat by using the yield return syntax:

All Bind does is instantiate the class and return it. The class gets passed the source IEnumerable and the function. The class implements IEnumerable<U> for its output, which means it must be able to return an object implementing IEnumerator. The easiest way to do that is have the class implement IEnumerator<U> as well. Then the IEmumerable can just return 'this'.

The class is completely lazy, so it doesn’t even get the source IEnumerator from the source IEnumerable until the first call of MoveNext.

The two fields, sourceEnumerator, and functionEnumerator are the state. The first can have a state of null, which is the state before we got the first value.

The object diagram for the program again shows three objects wired in a chain from right to left:

Bind just wires the IEnumerable interface. The IEnumerable GetEnumerator method then effectively wires the IEnumerator interface (in the same direction). So you might wonder if the IEnumerable interface could be considered redundant. We not just make Bind wire up the IEnumerator interfaces and dispensed with IEnumerable altogether? That would work, but I guess the reason IEnumerable exists is because IEnumerator is already implemented by many underlying library collections. When writing a new class that will support foreach, we need only provide a GetEnumerator method that simply returns the underlying collection instead of implementing the whole IEnumerator interface. However in our class above, this didn’t help because we had to implement the whole IEnumerator interface because we were recombining multiple collections.

The IObservable monad is the push version of the IEnumerable monad, sort of.

Once the flow of data begins, it is indeed pushed (source to destination). The data is pushed using the IObserver interface. But with the IObservable IObserver pair of interfaces, it the destination that initiates the transfer. The destination uses the Subscribe method in the IObservable monad to register to observe the data. This Subscribing is also what initiates the transfer in the source. Once a transfer is completed, another transfer can usually be started by unsubscribing and resubscribing. When used in this way, IObservable is sort of a pull programming paradigm when you consider which end initiatiates the data transfer.

Some writers equate IObservable with "asynchronous". However, a pushing interface like IObserver can be either synchronous or asynchronous. Data flows from the source object by calling a method in the IObserver interface, called OnNext. That method can execute synchronously all the way to the destination end of the chain, or it can return at any point along the chain, and the data flow can resume from that point at a later time, which is what we refer to as asynchronous.

Pull communications can’t be asynchronous or broken up in time, at least not in a straight forward way. It either requires blocking the thread (we don’t want to go there) or using a Callback, or using a Task or future object (which we covered earlier). The IEnumerator interface, being a pull interface, can only work synchronously. With IEnumerator, the destination end pulls data by calling a method. The function must execute synchronously all the way to the source otherwise it would return without a result.

The ability of push style programming to be either synchronous or asynchronous is a good reason to default to using it. It is the reason ALA defaults to using push. Sometimes there are good reasons to use pull, but where it doesn’t matter, we prefer push. So it is worth looking at the IObservable monad for comparing with ALA, even though IEnumerable monads tend to be more common in practice, but only because they work well for database queries (pull data from the database). IObservable is the closest for comparison with the common ALA programming paradigms.

Unlike the IEnumerable/IEnumerator pair of interfaces which go in the same direction, the IObservable/IObserver interfaces go in opposite directions. The IObservable interface goes from destination to source whereas the IObserver interface goes from source to destination to carray the data.

In the context of monads, the IObservable interface, being in the direction of destination to source, is the one that is used by Bind. IObserable is then used to wire and initiate the IObserver interface in the opposite direction. This is exactly what we did earlier with the IMaybe push monad.

In the context of ALA, it is a disadvantage to combine the 'wiring' and the 'start transfer' in the same Subscribe method call. In ALA we keep these two things separate because we want the code for these two things to be in two separate places. The wiring code represents a user story and so goes in a user story abstraction in the top layer. We wire up the entire program first and then set it running. The starting of a data transfer is a run-time event. It originates in the same layer, for example, from a button domain abstraction that is wired to it. However, because this is the IObservable monad and not ALA, the Subscribe method will do both functions - the wiring of the observer and then starting a single data transfer.

Another thing we will do, like we did for the deferred/push version of the Maybe monad, is compose Actions instead of Funcs. When an Action is called at runtime, it will be passed an object that implements IObserver. The action will use the IObserver to output directly instead of having a function that retirns an IObservable. This greatly simplifies the code in the Actions, which is what we want because these Actions are application code. Instead the Bind function will take on extra work. It needs to create an IObserver object to pass to the actions.

If you look at the SelectMany in the reactive extensions library for C#, you will see that it takes a Func. But there are two overloads. In one, the Func returns an IObservable object as expected. For the other, it returns an IEnumerable. It’s a shame that the second overload doesn’t take an Action that takes an IObserver. That would have truly simplified things. Anyway that’s what we will do in our example here.

Here is an action to use in our example applicaton:

It takes a single integer as input and outputs a stream of three integers. The output goes to the IObserver that is also passed to the Action.

Here is our top layer application code.

We start with a single integer with value zero. We conver it to IObservable using the reactive extensions Observable.Create method. Then we can use Bind on that to compose the action. We do that using the same action for all three times. Finally we send the output to the Console. We use an overload of Subscribe that creates a destination object.

Now let’s write the Monad’s bind function. As usual, C# (in this case the reactive extensions library) provides us with a shortcut way to implement Bind by using Observable.Create and Observer.Create. This shortcut method obscures the way the Bind function is a refactoring of the imperative code, which is our purpose. However, for reference, here is the shortcut version first:

If you find this version hard to read, just skip forward to the next version.

You may think we do not need the extra observer. Why not just pass outputObserver to the action like this:?

That would indeed correctly pass the multiple outputs of the action to the outputObserver. However, the action may call OnCompleted at the end of each of its sequences. If it does we need to intercept it and remove it because otherwise it will terminate the outputObservable sequence prematurely. This removal of the OnCompleted from the function’s output is effectively what 'flattens' the output.

Removing the OnCompleted call is the reason we use Observer.Create().

Now we do a version that does not use either Observable.Create or Observer.Create. Although the code is longer, this will be easier to understand since our purpose is to show how we can refactor the original imperative code. This shows more clearly that the Bind function works by instantiating an object that will do all the work at runtime, and then simply wires that object to the previous one.

The bind function simply instantiates an object from an explicit class called Observer. This class is listed below.

Finally, let’s do a deferred version of the mod360 monad that we used as one of our original examples. You’ll remember that we had imperative code that was doing mod 360 after every function call. We already did a simple immediate version of the monad. Let’s skip the deferred/pull version and go straight to the deferred/push version.

Here is a suitable interface for the monad:

Item0 in the Tuple is the angle, and Item1 in the tuple is the rotations.

And we will need a chaining interface for the Bind function to use:

Here is the application example code using the monad:

Application layer code

Here is the Bind function and ToMod360 function. Both use explicit classes to do the actual work.

Monad layer code

The Bind function just instantiates a Mod360 class, configures it with the function being composed, and wires it to the previous object using the Subscribe method of its observable interface. The Subscribe method effects wiring in the opposite direction using the observer interface, which is needed because it is a push monad.

The class that does the work for the Bind function is below. It implements IMod360Observer for use by the previous object, and IMod360Observable for use by the next object.

The Observer.Push function does all the work at runtime. It first calls the composed function, and then creates a result Tuple using the source Tuple and the Tuple that is returned by the function.

This is the class used by ToMod360, which is straightforward.

Note that previously with the IObservable monad, the Subscribe method in the IObservable interface had two function, to wire the IObserver interface, and to start the data being pushed from the source. I kept that behaviour because that how reactive extensions works. However I prefer that push programming paradigms are true push style.

So in this Mod360 monad, I have deliberately gone to a purely push paradigm. Calling the Subscribe method from the destination end does not intiate the dataflow. Instead I keep a reference to the source, which has a Run method. This makes an object structure that is more purely a push system, because the initaition of the dataflow is not done by a pull call from the destination end. This is much closer to how ALA works for its default programming paradigms. If you look at the top layer application code above, you will see that we kept a reference to the first object in the chain instead of the last. We called it program. To make the program run, we called program.Run().

That completes our four examples of refactoring imperative code using the monad refactoring pattern to defffered push versions of monads. We are now in a position to understand the general monad refactoring pattern.

In the examples of Bind above, the type that Bind takes and returns is generally a class or interface. A class is like an interface with only one implementation, so we are generally going to think of it as an interface. We did have one example where it was an integer, and one where it was a Func, but these too can be thought of an interface in a broad sense.

The interface can be anything we want for the refactored code to communicate along the chain. It can be an actual interface, such as IEnumerable<T>, or IMaybe<T>, or it can be a class such as Task<T>. Or it can be a complex interface that we write to get any common information we want through the chain.

Bind always takes this interface and returns the same interface. You can therefore chain Bind calls together using fluent syntax (dot operator).

The interface is usually generic, so takes a type as a parameter, e.g. Interface<T>. The Bind function takes an Interface<T> and returns an Interface<U>. So the generic type can change as it goes along the chain.

The pattern is about composing functions. These functions generally take a T and return an Interface<U>.

Here is an application that composes three functions using a Bind function:

When composing functions like this, you can’t explicitly see the type of the interface that’s being used. While debugging, I sometimes insert a decorator to write the type to the console like this:

Here is pseudo code showing the actual types:

As you can see, while Bind always takes an interface and returns the same interface, the generic type may change along the way. In our examples above we didn’t change the type much, but normally you can.

Here is a diagram of the monad pattern.

As you can see, monads are a 2-layer pattern. The two layers correspond roughly with ALA’s application and programming paradigms layers. The code that uses Bind to compose functions, and the lambda functions themselves are in the application layer. The Bind function and the Interface<T> are in the programming paradigms layer. Often monads come with a set of more specialized functions such as Sort, Filter and Sum. These would go in the equivalent of the domain abstractions layer. These functions either use Bind, or do the equivalent logic as Bind themselves.

The functions that are being composed take a T and return an Interface<U>. It is tempting to think that the Bind function simply returns the Interface<U> that is returned by the function, because they have the same type. But that is not usually the case. Bind usually creates a new object that implements Interface<U>, and then combines information from both the input Interface<T> and the output of the function to provide the output Interface<U>. That’s what the diagram is trying to convey.

In many explanations of monads, they call the interface the monad type, or a wrapped type, or a container type, or a type in a box, or an amplified type, or just the notation M T. I don’t think any of these terms are helpful in understaning monads. The wrapped, container and box terms don’t work well for deferred monads, which don’t actually contain a value. They contain a means of getting a value. For example, the deferred version of a list is IEnumerable. If our function returns an IEnumerable, that’s not really a container or box.

The term amplified just introduces another seemingly abstract concept which is unnecessary. And the term Monad type or the notation M T seems a bit circular - let’s not explain monads in terms of monads. So I prefer to think of the thing that the Bind function takes and returns as simply a chaining interface. It sometimes has one implementation, such as Task or List, but often it has more than one implementation such as IMaybe or IEnumerable. Usually the Maybe monad uses IMaybe with two implementations, one for when there is a value and one for when there is no value.

So generally I just think of it as Interface<T>.

The monad pattern requires three things: * an Interface<T> (the chaining interface) * a constructor or method for making ordinary values of type T into an object that implements Interface<T> * a Bind function that takes an Interface<T>, returns an Interface<U>, and is passed a function of the form Func<T, Interface<U>>.

The constructor or method for getting ordinary values into Interface<T> form is required to get started at the beginning of a chain.

For push style monads, we didn’t exactly follow this definition because the functions didn’t always return the chaining interface, but they did something equivalent.

The chaining interface and the Bind function can pipe through any extra information or capability we want through the interface. We could, for the sake of a silly example, pipe through an audio stream if we really wanted to. We could compose functions to modify the stream.

By using plain objects the barrier to understanding seems lower than for monads, at least for developers already familiar with objects. Functional programming, and monads in particular, seem to have quite a high barrier to entry unless you are a mathematician. The world needs the programmers who are able to understand objects but do not necessarily understand mathematical notation. I’m not sure what would happen if all universities only taught functional programming so that everyone is introduced to pure functions first. Perhaps then it would be objects which have a barrier to entry.

ALA’s domain abstraction objects are easier to understand than monads because they are plain objects. The mental model of composition in ALA is wiring instances of domain abstractions by their ports, which is conceptually just a component model. Monads compose functions so the mental model is primarily oriented to composing a chain of functions as a dataflow. To make an analogy with electronics, ALA is like composing ICs (integrated circuits with many pins with many functions) and monads is more like composing two-port components such as resistors, capacitors, inductors and transistors.

There seems to exist computing problems that are best described using state. Objects are the language feature that provides for this. Monads end up using objects with state anyway - they are just hidden beneath the covers.

The only slightly difference between ALA’s domain abstraction objects and plain object oriented objects is the use of ports. Port are used for all run-time input and outputs. Any programmer with familiarity with dependency injection can understand that a port is just an implementiion of dependency injection. A port is implemented simply as a field of the type of an interface, or is an implemented interface. As with normal dependency injection, the field is assigned a reference to another object that implements the port interface.

Unlike conventional dependency injection, the field is not assigned by the constructor or any setters. Instead the field is always assigned through use of WireTo or WireIn.

The difference between ALA and conventional dependency injection is that the interface used must be more abstract than either of the classes. It cannot be thought of as an abstract base class. It is even more polymorphic than that. This type of interface is called a programming paradigm, and can be implemented by many disparate classes. Therefore, the dependency injection cannot be container based. Instead the application code must explicitly instantiate the required objects and then wire them together.

Because ALA uses plain objects, and plain interfaces as their ports, ALA developers can add new domain abstractions and programming paradigms themselves. In the functional world, developers can certainly write new monad types, but it doesn’t seem that easy, and seems generally left to library developers. The abstraction level of these libraries is therefore generally not as close to the domain, and does not make a DSL. In ALA, the set of domain abstractions and programming paradigms that you write is a DSL.

Although ALA supports multiple programming paradigms, the dataflow programming paradigm is quite a common one. So we will have many domain abstractions like those that come with monad libraries like Select, Where and Sum. It is worth comparing the syntax of ALA using dataflows with monad syntax.

Here we are comparing the code in the top layer, the code that describes a user story. Both monads and ALA use fluent style with dot operators.Here is the syntax for monads:

And here is the syntax for ALA:

In the monad version, the Filter and Select functions do both the wiring and specify the operation to be wired, whereas in the ALA version these are kept separate. Keeping them separate has advantages that we will discuss shortly.

The ALA code can be generated from a diagram. However, there is nothing stopping us achieving exactly the same syntax as the monad version if we really want to. We just create extension methods that both create an instance of, and wire up, each abstraction such as Select and Filter:

The code for these extension methods would be located in the same abstractions as the Select and Filter classes respectively.

Note that IDataFlow is the type of the ports being wired. IDataflow is a push interface (similar to IObserver). So IDataFlow goes in the forward direction (the same direction as the data flows). The Select and Filter extension methods can’t be defined on IDataFlow. We need an interface on which to define Select and Filter. This interface must go in the reverse direction, from destination to source. So that’s what IChainable is for. IDataFlow and IChainable are analogous to the IObserver and IObservable interfaces respectively. Note, though, that IChainable only exists to give us an interface on which to define the extension methods. It doesn’t do anything else, so it contains no methods:

It doesn’t have a Subscribe method because the Select and Filter extension methods did the wiring up of the IDataFlow ports of the instances of the abstractions.

In ALA we keep the WireIn and new operators separate for the following reasons:

This is the second method of using monads in an ALA application.

It uses a domain abstraction that can be configured with a monad expression. This domain abstraction uses your normal ALA dataflow programming paradigm forits its input and output ports. We will do two versions, one configured with an IEnumerable chain and one configured with an IObservable chain.