Software Engineering - as it should be done

John R Spray

Last update: 2019-04-04

I would like to acknowledge the help of Roopak Sinha at AUT (Auckland University of Technology) in the writing of the paper for ECSA 2018 including his academic perspective.

ALA Paper presented at ECSA 2018

Organisation of this site:

Each chapter ends with an example project. These projects ground the ideas in real code. Unlike most pedagogical sized examples, these examples are progressively non-trivial. Yet because of ALA’s power, they remain small and easily readable in a few minutes.

1. Chapter one - What problem does it solve?

1.1. The Big Ball of Mud

ALA is a high level strategy to structure code. It tells you how to organise your code so it doesn’t degenerate, little by little, into sphagetti code, or what Brian Foote and Joseph Yoder describe as a "big ball of mud" during its life cycle.

Existing architectural patterns, styles, or principles (such as Layers, Decomposition, DSLs, Components, Models, Event-Driven, MVC, Inversion of Control, Functional Programming and Object Oriented Design, Single Responsibility) are insufficient by themselves. What is needed is a higher organisational strategy. That strategy makes it clear where, how and why to use all these other styles.

1.2. An optimal reference architecture

ALA is a reference architecture. It is independent of any specific functional domain, so it is a general reference architecture. The reference architecture is 'optimal' for certain non-functional requirements. By optimal, I mean that it makes these qualities as good as they can be.

  • Readability

  • Complexity

  • Maintainability

  • Testability

If other non-functional requirements are also important, ALA provides a good starting point. Even if the ALA structure must be compromised for other qualities, it is still better to start with these quality attributes optimised and deviate from them as necessary. As it happens, the maintainability resulting from ALA frequently makes other quality attributes easy to achieve as well, without significantly compromising these attributes. For example, in an ALA application it is often easy to make local performance optimizations in the interfaces that don’t affect the application code. Or, you can port an application without changing the application code.

1.3. Readability

close up code.jpg
Figure 1. Code quickly becomes a big ball of mud

ALA code is readable, not because of style, convention, comments or documentation, but because any one piece of code appears to you as a separate little program.

1.4. Complexity

There is a meme in the software industry that says that the complexity of software must be some function of its size. This need not be so. With proper use of abstraction it is possible to have complexity that is constant regardless of program size. ALA makes use of this.

complexity curve.png

This is a qualitative graph comparing the complexity of an ALA application with that of a big ball of mud and an average loosely coupled application. This is further explained later here.

1.5. Maintainability

The maintainability effort over time should qualitatively follow the green curve in the graph below because as software aritifacts are written, their reuse should reduce the effort required per user story. Product owners seem to have an innate sense that we manage to organise our code such that this happens, and that is why they get so frustrated when things seem to take longer and longer over time. In practice, too often we follow the red curve. Maintenance eventually gets so difficult that we want to throw it away and start again, thinking we can do better. My experience is that we never do better during the rewrite. It is just a psychological bias on the part of the developer caused by a combination of a) the Dunning Kruger effect and b) the fact that it is easier to read our own recently written code than someone elses.

If we apply all the well known styles and principles, the best we seem to be able to manage is the orange curve, which still has maintenance effort continuously increasing with an exponential factor.

effort curve.png

ALA is based on the theoretical architectural constraints needed to follow the green curve.

1.6. Domain oriented

As has been found useful in other methodologies such as Domain Specific Languages, Domain Driven Design, Model Driven Software Development and Language Oriented Programming, ALA provides a way to be 'domain oriented'.

But unlike most of the other domain oriented methodologies, ALA provides a way to be domain oriented with ordinary code, and with the same development envorinment. It is just a way to organise ordinary code to be domain oriented.

1.7. The software engineer’s trap

Typical bright young engineers come out of university knowing C++ or Java (or other C*, low-level, imperative, language that mimics the silicon), and are confident that, because the language is Turing-complete, if they string together enough statements, they can accomplish anything. At first they can. Agile methods only require them to deliver an increment of functionality. There hardly seems a need for a software architect to be involved. And besides, we are told that any design can emerge through incremental refactoring.

Cynefin.jpg
Figure 2. Code can quickly get complex

As the program gets larger, things are getting a little more complicated, but the young developer’s brain is still up to the task, not realizing he has already surpassed anyone else’s ability to read the code. He is still able to get more and more features working. One day the code suddenly 'transitions'. It transitions from the complicated quadrant into the complex quadrant. And now it is trapped there. It is too complex for the in-the-large refactoring that would be required to make it transition back. This pattern happens over and over again in almost all software.

The incremental effort to maintain starts to eat away and eventually exceed the incremental increase in value. This now negative return causes the codebase itself to eventually lose value, until it is no longer an asset to the business.

When a new bright young engineer who knows C* arrives, he looks at the legacy codebase and is convinced that he can do better. And the cycle repeats. This is the CRAP cycle (Create, Repair, Abandon, rePlace). ALA is the only method I know that can prevent the CRAP cycle.

1.8. The story

From early on in my career, I experienced the CRAP cycle many times. Each time I wanted to find a way to not fall into it. I would research and use all the architectural styles and principles I could find. I would come across things like 'loose coupling', and I remember asking myself, yes but how does one accomplish that?, and still fail.

I started searching for a pre-worked, generally applicable, 'template architecture' that would tell me what the organisation of the code should look like for any program. I searched for such a thing many times and never found one. Some would say that this is because the highest level structure depends on project specific requirements.

Forty years worth of mistakes later, I finally have that template meta structure that all programs should have. The turning point was when I noticed two (accidental) successes in parts of two projects. These successes were only noticed years later, 15 years in one case and 5 years in the other. They had each undergone considerable maintenance during that time. But their simplicity had never degraded and their maintenance had always been straightforward. It was like being at a rubbish dump and noticing two pieces of metal that had never rusted. "That’s weird", you think to yourself. "What is going on here?"

One of them had the same functionality as another piece of software that I had written years earlier. That software was the worst I had ever written. It was truly a big ball of mud, and maintenance had become completely impossible, causing the whole product to be abandoned. So it wasn’t what the software did that made the difference between good and bad. It was how it was done.

Analysing the common properties of those two code bases, gave clues that eventually resulted in a theoretical understanding of how to deal with complex systems. This meta-structure is what I now call Abstraction Layered Architecture.

Subsequently, I ran some experiments to see if the maintainability and non-complexity could be predictably reproduced. These experiments, which have worked spectacularly well so far, are discussed as a project at the end of every chapter.

1.9. Simplify the overwhelming software architecture styles, patterns & principles

Currently the problem of structuring software code to meet quality attributes involves mastering an overwhelming number of software engineering topics:

  • Complexity, Understandability, Readability, Maintainability, Modifiability, Testability, Extensibility, Dependability, Performance, Availability, Scalability, Portability, Security, usability, Fault-tolerance

  • Views, Styles, Patterns, Tactics, Models, UML, ADL’s, ADD, SAAM, ATAM, 4+1, Decomposition

  • CBD/CBSE, C&C, Pipes & Filters, n-tier, Client/Server, Plug-in, Microservices, Monolithic, Contracts, Message Bus

  • Modules, Components, Layers, Classes, Objects, Abstraction, Granularity

  • Information hiding, Separation of Concerns, Loose Coupling & High Cohesion

  • Semantic coupling, Syntax coupling, Temporal coupling, existence coupling, Dependencies, Interactions, Collaboration

  • Interfaces, Polymorphism, Encapsulation

  • Execution models, Event-Driven, Multithreaded, Mainloop, Data-driven, Concurrency, Reactor pattern, Race condition, Deadlock, Priority Inversion, Reactive

  • Principles: SRP, OCP, LSP, ISP, DIP; MVC, MVP, etc

  • Design Patterns: Layers, Whole-Part, Observer, Strategy, Factory method, Wrapper, Composite, Decorator, Dependency Injection, Callbacks, Chain of Responsibility, etc

  • Expressiveness, Fluency, DDD, Coding guidelines, Comments, Documentation

  • Programming Paradigms, Imperative, Declarative, OO, Activity-flow, Work-flow, Data-flow, State machine, GUI layout, Navigation-flow, Data Schema, Functional, Immutable objects, FRP, RX, Monads, AOP, Polyglot-Programming Paradigms

  • Messaging: Push, Pull, Synchronous, Asynchronous, Shared memory, Signals & Slots

  • Memory management, Heap, Persistence, Databases, ORMs

  • Up-front design, Agile, Use cases, User stories, TDD, BDD, MDSD

Mastering all these topics takes time. Even if you can, juggling them all and being able to use the right ones at the right time is extremely taxing on any developer. Add to that the mastering of technologies and tools, keeping to agile sprints deadlines, and commitment to your team and management, it is an almost impossible task. 'Working code' tends to be what the team is judged on, especially by project managers or product owners who have no direct interest in architecture or even the Definition of Done. They don’t want to know about the rather negative sounding term, "technical debt".

ALA works by pre-solving most of these software engineering topics into a single 'meta-style' This meta-style provides a simple set of architectural constraints.

Being a pre-worked recipe of the aforementioned list of styles and patterns, ALA contains no truly novel ideas. Some ingredients are accentuated in importance more than you might expect (such as abstraction). Some are relatively neutral. Some are putposefully left out. The biggest surprise for me during the conception process of ALA was that some well-established software engineering memes seemed to be in conflict. Eventually I concluded that they were in-fact plain wrong. We will discuss these in detail one at a time in subsequent chapters. But to wet your appetite here is one meme that ALA throws out: the UML class diagram. Read on to find out why.

Like any good recipe, the ingredients work together to form a whole that is greater than the sum of parts. The resulting code quality is significantly ahead of what the individual memes do. It continues to surprise me just how effective it is.

1.10. The first few strokes

As a software engineer contemplating a new project, I have often asked myself "Where do I start?" This also happens with legacy code, when contemplating the direction that refactorings should take. "If this software were structured optimally well, what would it look like?"

Christopher Alexander, the creator of the idea of design patterns in building architecture, said, "As any designer will tell you, it is the first steps in a design process which count for the most. The first few strokes which create the form, carry within them the destiny of the rest". This has been my experience too.

In Agile, where architecture is meant to emerge, this wisdom has been lost. ALA restores that wisdom to software development, and gives the software architect the exact process to follow for that little piece of up-front design. No more than one sprint is required to do this architectural work, regardless of the size of the project.

Furthermore, once this architectural work is done, the Agile process works significantly better thereafter. Furthermore, my experience over several projects so far is that the initial architecture does not need to change as the development proceeds.

1.11. Example project - Functional composition

In this example, we use 'functional composition' because it is a programming paradigm we all already know. However, keep in mind that simple functional composition (without monads) is not a suitable programming paradigm for the most part of a typical programs. It suits when a problem requires dedicated CPU to process a job as fast as it can in computer time, the sequence is known ahead of time (proactive not reactive), nothing else needs doing while this is happening, and it doesn’t have to wait for anything while it is being done. Nevertheless, this can be a solution for small programs.

Applying ALA to functional composition means three things:

  • Every function is an abstraction.

For our purpose here, an abstraction means that our brain can easily learn (by reading the function name or a comment) and retain what a function essentially does. It means that when other programmers are reading your code where a function is called, they don’t have to 'follow the indirection' - they can stay with the code unit they are in and read it like any other line of code. It means a single responsibility. It means it knows nothing about the content of any other abstractions. It means reuseable, and it means stable. The name of the function should not be generic ProcessData, or CalculateResult. It should not be the name of the event that caused it to be executed like PulseComplete.

  • Functions go in a small number of discrete abstraction levels.

This implies that function call depth is at most three (not counting library functions at a 4th level).

The first level function contains all knowledge about the application requirements. No implementation here, just describe the requirements in terms of other functions.

The second level is functions that contain knowledge about the domain. It has all the abstractions needed to make it possible for the first level to describe the requirements. No function at this level knows anything about the specific application. An example would be calculate mortgage repayments, or filter data.

The third level functions are at an even greater level of abstraction, things that would be potentially reusable in many domains. It should have the abstraction level of the types of programming problems being solved. Examples might be communications, persistence, logging. None of these functions can have any knowledge of the specific application, nor the domain. So the persistence functions are not persistence of specific domain objects. With configuration, they would know how to persist anything.

A function that doesn’t clearly belong at one of these abstraction levels should be split in two. Specific application knowledge gnerally becomes configuration parameters in the higher layer of a more abstract function in the lower layer.

For completeness, a 4th level would be your programming language library. No where in these levels is the underlying hardware, nor data. Later we will see where they go, but for now forget all preconceived notions of layers such as UI, business logic and Database. In ALA, these are incorrect.

  • The first layer just describes the requirements.

The top layer describes requirements and that’s all it does (like a DSL). It composes functions from the lower layers, and configures them for a specific purpose according to the requirements.

Let’s look at some bad code that breaks each of these constraints and then the corresponding code that fixes them.

1.11.1. Bad code

main.c
 void main()
 {
    while (1)
    {
        GetTemperatures(temperatures); // gets 100 temperatues (1)(2)
        temperatures = ProcessTemperatures(temperatures); (1)(2)
        Display(temperature);
    }
 }
process.c
 // do everything needed to process an adc reading
 float ProcessTemperatures(temperatures)  (4)(5)
 {
    for (i = 0; i<100; i++) {  (3)
        temperature = (temperatures + 4) * 8.3;  (3)
        rv = Smooth(temperature);  (6)(7)
    }
    return rv;
 }
smooth.c
 // smooth the reading before displaying
 float SmoothTemperature(temperature) (4)
 {
    static filtered = 0;
    filtered = filtered*9/10 + temperature/10; (3)
    return filtered;
 }
1 function name is specific to this application, destroying it as a potential abstraction
2 functions are collaborating to implement the 100 samples at a time requirement
3 details from requirements appearing inside functions (all the constants), destroying potential abstractions
4 function name doesn’t describe an abstraction
5 function has three responsibilities, process 100 samples at a time, convert to Celsius, and Filtering
6 function composition in wrong level (only the application knows this needs doing
7 function composition too deep (function composition should be shallow)
8 Temporal problems - if adc readings take 1 ms, main loop time is 100 ms

1.11.2. Better code

application.c
 void main() (1)
 {
    while (1)
    {
        adc = GetAdcReading();  (2)
        temperatureInCelcius = OffsetAndScale(adc, offset=4, slope=8.3); (5)
        smoothedTemperature = Filter(temperatureInCelcius, 10); (6)
        if (SampleEvery(100)) (4)
        {
            Display(smoothedTemperature)
        );
    }
 }
offsetandscale.c - (domain abstraction)
 // offset and scale a value
 void OffsetAndScale(data, offset, scale) (3)
 {
    return (data + offset) * scale;
 }
filter.c - (domain abstraction)
 // IIR 1st order filter, higher filterstrength is lower cutoff frequency
 float Filter(int input, int filterStrength)  (3)
 {
    static float filtered = 0.0; (7)
    filtered = (filtered * (filterStrength-1) + input) / filterStrength
    return filtered;
 }
sample.c - (domain abstraction)
 // Returns true every n times it is called
 bool SampleEvery(int n)  (3)
 {
    static counter = 0; (7)
    counter++;
    if (counter>=n)
    {
       counter = 0;
       rv = true;
    }
    else
    {
       rv =  false;
    }
    return rv;
 }

The code is now arranged clearly into two abstraction layers, the application layer and the domain abstractions layer. Other domain abstractions used by the application are not shown: the ADC, and the Display.

1 The application is readable on its own without having to go and read code inside any of the abstractions it uses.
2 The application describes the requirements, all of the requirements, and does nothing else. It delegates all the actual work to domain abstractions. Only the application knows it is a thermometer. The application knows nothing of how the abstractions work, only what they do.
3 None of the abstractions know anything about each other or anything about the application. They don’t know they are being used to make a thermometer.
4 The application knows the requirement detail of how many ADC readings are needed for each temperature display update.
5 Application knows conversion factors from ADC to Celcius but not how to do offsetting or scaling.
6 Application knows the configuration of the filtering it needs but not how to do filtering.
7 The emphasis is on abstraction not zero side effects. Filter and SampleEvery are good abstractions despite having a side effect.
8 Temporal problems mitigated somewhat. Main loop time still includes the ADC conversion time and Display output time.

Also note other things that can be done in the abstraction layered version.

  • The application can easily be modified (rewired) to say swap the order of processing of the scaling and the filtering, or insert a new data processing operation between say the scaling and the filter, or adding a logging output destination at a higher data rate, switching to a different type of ADC.

  • No domain abstractions would change to make any changes to the thermometer. This is because as abstractions, they don’t know anything about the specific application.

  • In this functional composition, data comes into the application code. That’s why the application has some local variables to identify the data at various points during the processing. In most other programming paradigms we will use, the data will go directly between instances of the abstractions at run-time, not come up into the application. The application will then only instantiate and wire together instances of abstractions at the start, not handle the actual run-time data.

1.11.3. Future glimse

Remember that while instances of abstractions can be composed perfectly well with functional composition, it is not generally a useful programming paradigm or execution model. We have used imperative functions here only because they are a very familiar programming paradigm that we all learnt in our first programming language. But we won’t get far doing everything with functions. The problem is that functional composition forces the execution flow to follow exactly the composition flow. This only suits a narrow range of problems. Usually we will need to use ALA in other programming paradigms (execution models) that separate execution flow from composition flow.

For example, the most common programming paradigm we will use is Data-flow - when we compose domain abstractions together, we mean that at run-time data will pass between adjacently wired instances. A common Data-flow paradigm is monads. This is what the Thermometer example would look like using them.

application.c
 void main()
 {
    program = new ADC()
    .OffsetAndScale(adc, offset=4, slope=8.3)
    .Filter(10)
    .SampleEvery(100)
    .Display();

    program.Run();
 }

Don’t worry if you don’t know what monads are - you don’t need them to understand ALA. They are only being used here as an example of data-flow. Look how the execution loop has somehow been abstracted away, leaving just a desciption of the data-flow that describes the requirements. It no longer specifies the execution flow, which abstracted away.

Here is another quick glimpse of how you could compose the Thermometer, but this time using plain old classes instead of monads.

application.c
 void main()
 {
    program = new ADC()
       .wireIn(new OffsetAndScale(adc, offset=4, slope=8.3))
       .wireIn(newFilter(10))
       .wireIn(new SampleEvery(100))
       .wireIn(temperature = new NumberDisplay());

    mainwindow = new Window()
       .wireTo(new Label("Temperture:"))
       .wireTo(temperature);

    mainwindow.Run();
 }

This time we are wiring together instances of abstractions in a more general way. There are multiple programming paradigms here - the meaning of the wiring is data-flow in some parts, and UI layout in other parts. This is all done in the one coherent application.

The examples to follow will use a range of different programming paradigms and consequently 'composition' will mean different things. Sometimes we will use custom programming paradigms - whatever allows us to describe those requirements in the best way.

2. Chapter two - What does the structure look like?

In this section we just describe the anatomy of the ALA structure without trying to explain too much about why it looks that way.

We describe it in several different ways because we all have different experiences, different 'hooks' on which we hang new ideas. So we need different perspectives to 'get' an idea. Find the one that best explains the insight for you and don’t worry about the rest.

2.1. Code organisation into folders

2.1.1. Read this if you need to read an existing ALA application.

We start with a practical viewpoint of ALA - how it organises code into folders.

If you see an ALA application, you will find three folders called:

  • Application

  • DomainAbstractions

  • ProgrammingParadigms

There should also be a readme file that points to this website (or equivalent documentation). In ALA, we are explicit about what knowledge is needed before something can be understood (knowledge dependencies). To understand an ALA structured application, you need a basic understanding of ALA (from this chapter). So that’s why there should be a readme file.

Continuing with the idea of knowledge dependencies, the class in the Application folder will have knowledge dependencies on the classes in the DomainAbstractions folder. In other words, you need to know what the classes in the DomainAbstractions folder do in order to read the application code. Similarly the classes in the DomainAbstractons folder have knowledge depedencies on the interfaces in the ProgrammingParadigms folder. There are no dependencies between classes within a folder.

In the Application folder, you will usually find a diagram. This diagram describes the requirements. The diagram is 'complete' in that it describes all details of the requirements - it is not just an overview. The diagram is itself 'executable'.

It should be quite easy to read the diagram as it only describes the requirements and does not involve itself with implementation. The boxes are instances of the DomainAbstractions (objects). The lines are connections between the objects.

There should be a code file that exactly represents the diagram. It is generated from the diagram. So the diagram is the source. However, looking at this code file may clarify how the diagram is represented in code.

Every box in the diagram is an instance of one of the classes in the DomainAbstractions folder. These classes are called abstractions because they have zero knowledge of each other, and definitely know nothing about the application. Their abstraction level is more general than the application, and so they are reusable within a domain. For now a domain can just mean your company.

The lines in the diagram represent connections using one of the interfaces from the ProgrammingParadigms folder. There is usually more than one interface, but no more than a few. Each represents a 'programming paradigm' such as event flow, data flow, a UI composition, or a schema relationship. The abstraction level of the ProgrammingParadigms folder is more general again than the DomainAbstractions - each paradigms should be useful for a type of computing problem in many different domains. This is the 'abstract interactions' pattern.

This small set of interfaces allows instances of domain abstractions to be wired together in an infinite variety of ways.

2.2. How classes are used

this is another practical viewpoint, this time on how classes are used in ALA programs.

In ALA, a class’s public interface (it’s public methods and properties) are only used to instantiate and configure the class. It is not used for anything the class actually does. The public interface is 'owned' by the class so is specific to what the configuration of that class. The public interface is only used from a class in a layer above. Only that layer knows what should be instantiated, how it should be configured, and how the instantiated objects fit together to make a system. This is a composition relationship.

All other operations are done through interfaces. Class don’t 'own' these interfaces - they are not specific to any one class. They are not about what any one class does, or needs. They are more general so that typically many different classes will implement/accept them. Objects of different classes can then be connected together using these more general interfaces. The implication is that classes do not have association relationships. The lines that you would normally see dominating most UML class diagrams are completely absent if you drew an class diagram of an ALA application.

ALA doesn’t need or use inheritance either. So the only relationship between classes is composition. If you drew a class diagram in ALA, you wouldn’t draw lines for composition. This is because you are composing abstractions. You wouldn’t draw a line to a square-root function every time you used it. It’s the same thing when using any abstraction. So it turns out that if you did try to draw a class diagram in ALA, it would have no lines at all. So there’s no point.

Any given class will typically implement/accept more than one of the generic interfaces - one for each object it might be connected to in a diagram. Think of them as I/O ports. This is the interface segregation principle, except that we do not refer to the other objects as clients. Only the class in the layer above (that uses the public interface) has the status of a client. The objects to which an object is wired are peers.

2.3. Abstraction Layers

In contrast to the previous two sections that talk about the use of folders and classes, this section gives the most abstract perspective we will use. I introduce it now because it is the one that gives ALA its name.

The diagram shows the abstraction layers:

Layers diagram
Figure 3. The four ALA layers

The first problem in understanding abstraction layers is understanding what abstraction means. Unfortunately the software industry has misused the word to the point where we get things upside down. This comes about because it sees hardware or alternatively the database at the bottom, and since hardware and databases are 'concrete', it must be the least abstract. And so we build things on top of those that must get more abstract. Whatever is at the top, being the farthest away from the concrete silicon, must be the most abstract. In our case the top layer is the application. This thinking is completely wrong. We will look in depth at what 'abstract' means in a later section, but for now, just suspend everything you think you know about abstraction. In ALA we will say that 'more abstract' means 'more ubiquitous', 'more reusable' and 'more stable'. The application, at the top, is the least abstract. Also suspend everything you think you know about layers. In ALA, the hardware is never at the bottom. And neither is the database. Your programming language is.

Because this perspective probably doesn’t really connect with anything you already do, we will just list three key takeaway points from this section. These will become clearer later. In ALA:

  1. The only dependencies you are allowed are on abstractions (shown as green arrows on the diagram) and referred to as 'knowledge dependencies' (as opposed to run-time dependencies).

  2. The first three layers are Application, Domain Abstractions, and Programming Paradigms.

  3. As you go down the three layers, the abstractions get more abstract, and therefore more ubiquitous, more reusable, and more stable.

2.4. Executable Description of Requirements

If I had just two minutes to explain what ALA is, this is the perspective I would use:

This perspective puts the focus on your input information - the requirements. ALA is a methodology that directly describes requirements. The description of the requirements is executable. Instead of having two documents, one for requirements capture and one for software source, ALA combines has a single document and a single source of truth. BDD (Behavioural Driven Design) does something similar, but only achieves it for requirements and their tests. ALA goes one step further and makes the expressed requirements also the executable solution.

No Separate Architecture

The executable description of requirements in the top layer is also the architecture or the design. (I do not make a distinction between architecture and design.) There is no separate architecture, no model, no "high level" design. The same artefact that describes the requirements and is executable is also the application’s architecture. One source of truth for everything.

2.5. Create and Compose

If I had ten minutes to explain what ALA is, this is the perspective I would use.

A common cliché for tackling complexity is "divide and conquer". Now here is a surprise. In ALA we do not divide and conquer. Instead we use a different cliché, "Create and Compose"

Here are a few examples of conventional composition:

  • When we write code in a general purpose programming language, we are composing with statements. Statements are low level (fine grained) and only support a single programming paradigm, which we could describe as 'imperative' or 'sequential execution flow'. The structure is linear or a tree.

  • In functional programming, we are composing with functions, so the elements are higher level things that you create. But the programming paradigm is still sequential execution flow. The structure is linear or a tree.

  • When programming with monads, we are composing with amplified data types. These are usually low-level elements. But the programming paradigm is data-flow. The structure is usually linear. See my method to understand Monads in Chapter Six

  • When programming with UML class diagram, we are composing high-level classes. The programming paradigms are associations. The syntax is graphical. The structure is a network.

  • When programming with XAML, we are composing with fundamental UI elements. The programming paradigm is UI layout.

Let’s list the different properties present in these composition methods:

  • low-level/high-level - A fixed set of fundamental elements versus elements that you create.

  • Programming paradigm: The meaning of a composition relationship is fixed in each case. It can be Imperative, Data-flow, UI layout etc. But choose one.

  • Linear/Tree/Network: The structure built by the composition relationships can be linear, a tree structure or a more general network.

  • Syntax: The syntax for the composition relationship can be using spaces, dots or lines and we can use various types of bracketing or indenting for the tree structures.

In ALA, we are setting up the top layer so we can do composition that

  • Composes high-level elements that you create.

  • Uses a mix of all programming paradigms, and allows new ones that you can create.

  • Uses the same syntax for all composition relationships.

  • Easily allows linear, tree or network structures

ALA can be described as 'generalised create and compose'.

Generally, compositions are 'instances of abstractions' 'connected' together in a specific combination. This can be thought of as a graph. A graph is most easily imagined as a box and line drawing. In the common examples of composition that we mentioned above, sequential execution flow, monads, UI layout etc, the composition readily supports graphs that are linear or small tree structures. Arbitrary graph structures can usually be done by adding connections in a special way - by naming some of the nodes and then connecting by name. They is somewhat inconvenient and ununreadable in text form. ALA can use diagrams to allow all compositions to be arbitrary graphs.

To support generalized composition, ALA dedicates the top layer to the composition itself, a layer below it for the abstractions from which instances can be composed, and a layer below that for the different types of composition paradigms. This bottom layer is usually plain old interfaces.

2.6. No decomposition

In the previous section, we briefly mentioned that in ALA we don’t use divide and conquer, but use create and compose instead.

In this section, lets have a look at what is wong with divide and conquer.

Consider this phrase, often found near definitions of software architecture.

"decomposition of a system into elements and their relations".

Notice the word 'their', which I have italicised to emphasis that the relations are inferred to be between the said elements. It implies that the elements know something about each other. It implies they collaborate. This is a really bad meme. ALA is the antithesis of this meme.

Here is how to reword the meme for ALA:

"abstractions and composition of their instances".

Strictly speaking the wording of the decomposition meme does not preclude this meaning, but it is at best misleading. This seemingly subtle shift causes a complete change in the structure, as described in the two contrasting diagrams below:

2.6.1. The problem with "Decomposition into elements and their relations"

An architecture based on decomposition generally turns out something like this:

Decomposition into elements and their relations
Figure 4. Decomposition into elements and their relations

The figure shows five modules (or classes) and their relations (as interactions). Study almost any piece of software, and this is what you will find (even if it adheres to the so-called layering pattern).

The structure generally can be viewed as 'clumping'. Like galaxies, certain areas have higher cohesion, and so go inside boxes. Other areas are loosely coupled, and so are represented by lines between the boxes. The difference between high cohesion and loose coupling is only quantitative.

Software health in this type of architecture is effectively management of the resulting coupling between the boxes. Allocate code to boxes in such a way as to minimize coupling. This dependency management has two conflicting forces. One is the need to have interactions to make the modules work as a system. The other is to minimize the interactions to keep the modules as loosely coupled as possible. As maintenance proceeds, the number of interactions inevitably increases, and the interfaces get fatter. The clumping is gradually. The encapsulations are more or less transparent.

Various architectural styles are aimed at managing this conflict. Most notably:

  • layering pattern

  • MVC pattern

  • Dependency rules

    1. Avoid circular dependencies.

    2. Avoid high fan-in and high fan-out on a single module.

    3. Avoid dependencies on unstable interfaces.

Note that none of this 'dependency management' really avoids circular coupling. To some extent there will always be 'implicit coupling' in both directions between modules of a decomposed system. This is because the modules are the opposite of abstractions - specific 'parts' designed to interact and therefore collaborate. For example, a function of a decomposed system will tend to be written to do what its caller requires even if there is no explicit dependency on its caller. So circular dependencies may be avoided at compile-time, but will still be present at design-time. That is why in the diagram above, dependencies are drawn from the insides of each of the modules in both directions. This indicates that the code inside has some inherent design-time collaborative coupling. To the compiler or a dependency graphing tool, it some of the lines may appear to be in one direction only.

2.6.2. Abstractions and composition of their instances

When you use abstractions instead of modules, the qualitative difference is that there are no interactions, no collaboration, no dependencies between your abstractions at all:

Abstractions do not interact
Figure 5. Abstraction do not interact

The word 'modules' has been changed to the word 'abstractions'. All the dependencies are gone. And with them all their problems, and all their management. The implicit coupling that we talked about earlier is also gone. It no longer has a 'clumping' structure. Loose coupling is replaced with zero coupling.

The obvious question now is how can the system work? Where do all the interactions we had before go? The answer is they become normal code, completely contained inside one additional abstraction:

Abstractions and composition of their instances
Figure 6. Abstractions and composition of their instances

Interactions or collaboration should never be implemented in your abstractions. That just destroys them as abstractions. They are implemented inside another new abstraction at a different, more specific, abstraction level. Being inside that new abstraction the interactions are not dependencies. They are no longer dependencies because they are no longer relations between classes. They are just a composition of instances in a cohesive piece of code - the application. That code has all the knowledge that used to be collaborative knowledge between modules. In fact that’s all that code does.

Software engineering should not be about managing dependencies.

It should be about inventing abstractions.

2.7. DSLs

ALA’s succinct expression of requirements in the top layer sounds similar to the way requirements might be represented in a DSL (Domain Specific Language). Under the broader definition of a DSL, ALA’s domain abstractions layer is a DSL. But ALA is also different from a DSL. ALA, as its name suggests, is fundamentally about layering of abstractions. It layers them in a small number of layers, according to their abstraction level. When you do this, the top two layers emerge as the specific application and the domain level. Therefore ALA happens to converge on the same solution as DSLs for these top two layers.

In coming to this same solution from a different direction it has a different emphasis than a DSL has. It does not pursue the idea of an external DSL (new syntax), nor even the syntactic elegance of DSLs. It doesn’t move application development away from the developer as DSLs do. You don’t get a different language such as XAML that a UI specialist designer can learn. These things may still be desirable qualities and ALA does not preclude them, it is just not what ALA is about. ALA says that just getting the abstraction layering right is enough to deal with complexity and reducing effort of maintainability.

As a DSL, in ALA you usually just wire together plain old objects, or functions. Their classes (the domain abstractions) and their 3rd layer interfaces collectively form the DSL. The grammar is defined by which classes use which interfaces. This sets the rules for what types of objects can be wired together.

By the way, ALA also emerges other already discovered architectural styles such as CBE (Component Based Engineering), and compositionality, and these are discussed later.

2.8. SMITA (Structure Missing in the Action)

The problem in most large code bases is that the system structure is not explicit. It is distributed inside the modules themselves. If there is any collaboration between modules, it is implicitly hidden inside them. Finding this structure, even for a single user story can be time consuming. I have often spent a whole day doing that, doing countless all-files searches, just to change one line of code. Many developers I have spoken to can identify with this experience.

It can get a lot worse as the system gets larger. In a bizarre twist, the more loosely coupled you make the elements, the harder it gets to trace a user story through them (because of the indirections). Some people conclude that loose coupling and being able to trace through a user-story are naturally in conflict.

I call this situation SMITA (Structure Missing in the Action). This hidden structure is sometimes partially brought out as a model, a sort of high-level documentation of the internal structure. But such models are a secondary source of truth.

ALA completely eliminates this conflict. The structure of a user story is explicitly coded in one place, without any indirections. Yet the abstractions are zero-coupled.

2.9. Diagrams vs text

Sometimes diagrams and text are thought of as equivalent representations and it is a matter of personal preference which to use in a given situation. In ALA diagrams and text have two distinct uses in the top layer. Text is used when the relationship structure between the instances of abstractions is linear or a shallow tree. Diagrams are used when the relationship structure of the instances is an arbitrary network or graph, or a deep tree.

Use of diagrams in the second scenario is important to the design process, just as it is for and electronics engineer who would use a schematic diagram.

2.10. Real world metaphors

2.10.1. Atoms and molecules

Here are two atom abstractions: Oxygen atom Hydrogen atom

Instances can be composed to make a molecule: Water molecule

If water was implemented in the same way we typically write software, there would be no water molecule, and the oxygen molecule would be modified to instantiate hydrogen atoms and interact with them. Even if dependency injection is used to avoid the instantiating, it is still unlikely that a water abstraction would be invented to do that, and there would still be the problem of the oxygen module being modified to interact with hydrogen’s specific interface. Either way, the oxygen module ends up with some implicit knowledge of hydrogen. And hydrogen probably ends up with some implicit knowledge of oxygen in providing what it needs.

This implicit knowledge is represented by the following diagram. The relationship is shown coming from the inner parts of the modules to represent implicit knowledge of each other.

diagram o h.png

While oxygen and hydrogen are modules, they are not abstractions because oxygen is implicitly tied to hydrogen and vice-versa. They can’t be used as building blocks for any other molecules.

To keep the oxygen as abstract as it is in the real world, an interface must be conceived that is even more abstract than oxygen or hydrogen. In the molecule world this is called a polar bond.

The corresponding software would look like this:

Slide15.jpg

The water molecule has a "uses instances of" relationship with the two atoms, and the atoms have a "uses instance of" relationship with the even more abstract polar bond. Polar bond is an example of what we call an 'abstract interaction'.

2.10.2. Lego

The second real world metaphor is Lego. Shown in the image below is the same three layers we had above for molecules, atoms and bonding types.

Slide16.jpg

The domain abstractions are the various lego pieces, instances of which can be assembled together to make things. Lego pieces themselves have instances of an abstract interface, which is the stud and tube. There is a second abstract interface, the axle and hole. We also call the abstract interface the 'execution model' and here with the lego metaphor we start to see why it can be thought of in this way - when the model runs, the axle and hole interface executes.

2.10.3. Electronic schematic

The third real world metaphor comes from electronics. The abstractions are electronic parts, instances of which can be composed as a schematic diagram:

Slide17.jpg

In this domain, the abstract interfaces (execution models) are both digital signals and analog voltage levels.

2.10.4. A clock

The forth and final real world metaphor is a clock. In this diagram, we show the process of composition of abstractions to make a new abstraction. The process is a circle because instances of the new abstraction can themselves be used to make still more specific abstractions. Each time around the circle adds one layer to the abstraction layering.

Slide18.jpg

To go round the circle once, we start with abstract parts such as cog wheels and hands. Instances of these have abstract interfaces that allow them to interact at run-time, such as spnnin on axles and meshing teeth. The next step is to instantiate some of these abstractions and configure them. For example, configure the size and number of teeth of the cog wheels. Next comes the composition step, where they are assembled. Finally we have a new abstraction, the clock. Instances of them can be used to know when to do things during your day, but that is a whole different abstraction.

There are many other instances of this pattern in the real world, and in nature. In fact almost everything is composed in this way.

2.11. Example project - Ten-pin bowling

The ten-pin bowling problem is a common coding kata. Usually the problem presented is just to return the total score, but in this example project we will tackle the more complicated problem of keeping the score required for a real scorecard, which means we need to keep all the individual frame ball scores:

bowling scorecard2.png

The ALA method starts by "describing the requirements in terms of abstractions that you invent". When we start describing the requirements of ten-pin bowling, we immediately find that "a game consists of multiple frames", and a "frame consists of multiple balls". Let’s invent an abstraction to express that. Let’s call it a "Frame". Instances of Frame can be wired together by a "ConsistsOf" relationship. So let’s invent an abstract interface to represent that, and call it 'IConsistsOf'.

Here is the diagram of what we have so far.

diagram bowling 1.png

This is the first time we are using a diagram for an ALA application, so lets go through the conventions used.

The name in the top of the boxes is the abstraction name. The name just beneath that is the name of an instance of the abstraction. For the bowling application above, we are using two instances of the Frame abstraction, one called "game" and one called "frame". Below the abstraction name and instance name go any configuration information of the instance.

The Frame abstraction is configured with a lambda function to tell it when it is finished. The Frame abstraction works like this - when its last child is complete it will create a new one. It will stop doing that when the lambda expression is true. It will tell its parent it is complete when both the lambda expression is true and its last child Frame is complete.

The end of the chain is terminated with a leaf abstraction that also implements the 'IConsistsof' interface called 'SinglePlay'. It represents the most indivisible play of a game, which in bowling is one throw. Its job is to record the number of pins downed.

The concept in the Frame abstraction is that at run-time it will form a composite pattern. As each down-stream child frame completes, a Frame will copy it to start a new one. This will form a tree structure. The "game" instance will end up with 10 "frames", and each frame instance will end up with 1, 2 or 3 SinglePlays.

Note, in reference to the ALA layers, this diagram sits entirely in the top layer, the Application layer. The boxes are instances of abstractions that come from the second layer, the Domain Abstractions layer. The arrows are instances of the programming paradigm, 'InConsistsOf', which comes from the third layer, the ProgrammingParadigms layer.

This diagram will score 10 frames of ten-pin bowling but does not yet handle strikes and spares. So lets do some 'maintenance' of our application. Because the application so far consists of simple abstractions, which are inherently stable, maintenance should be possible without changing these abstractions.

The way a ten-pin bowling scorecard works, bonuses are scored in a different way for the first 9 frames than for the last frame. In the first nine frames, the bonus ball scores come from following frames, and just appear added to the frame’s total. They do no appear as explicit throws. In the last frame, they are shown as explicit throws on the scorecard. That why there are up to 3 throws in that frame.

To handle the different last frame, we just need to modify the completion lambda expression to this.

frameNum<9 && (balls==2 || pins==10) // completion condition for frames 1..9
|| (balls==2 && pins<10 || balls==3) // completion condition for frame 10

To handle the first 9 frames, we introduce a new abstraction. Lets call it Bonuses. Although we are inventing it first for the game of ten-pin bowling, it is important to think of it as a general purpose, potentially reusable abstraction.

What the Bonus abstraction does is, after its child frame completes, it continues adding plays to the score until its own lambda function returns true.

The completed ten-pin bowling scorer is this:

diagram bowling 2.png

Note that the "game" instance (the left box of the diagram) implements IConsistsOf. This is where the outside world interfaces to this scoring engine. During a game, the number of pins knocked down by each throw is sent to this IConsistsOf interface. To get the score out, we would call a GetScore method in this interface. The hard architectural work is done. We have invented abstractions to make it easy to express requirements. We have a diagram that describes the requirements. And the diagram is executable. All we have to do is put some implementation code inside those abstractions and the application will actually execute.

First lets turn the diagram into equivalent code. At the moment, there are no automated tools for converting such diagrams to code. But it is a simple matter to do it manually. We get the code below:

private IConsistsOf game = new Frame("game")
    .setIsFrameCompleteLambda((gameNumber, frames, score) => frames==10)
    .WireTo(new Bonus("bonus")
        .setIsBonusesCompleteLambda((plays, score) => score<10 || plays==3)
        .WireTo(new Frame("frame")
            .setIsFrameCompleteLambda((frameNumber, balls, pins) => frameNumber<9 && (balls==2 || pins[0]==10) || (balls==2 && pins[0]<10 || balls == 3))
            .WireTo(new SinglePlay("SinglePlay")
    )));

All we have done is use the 'new' keyword for every box in the diagram. We have made the constructor take the instance name as a string. (This name is not used except to identify instances during debugging.) We use a method called "WireTo" for every line in the diagram. More on that in a minute. And we pass any optional configuration into the instances using setter methods. The WireTo method and the configuration setter methods all return the 'this' pointer, which allows us to write this code in fluent style. If you are not familiar with fluent style it is just making methods return the this reference, or another object, so that you can chain together method calls using dot operators.

Not all ALA applications will be put together using the method in the previous paragraph, but I have found it a fairly good way to do it for most of them, so we will see this same method used other example projects to come.

So far, this has been a fairly top-down, waterfall-like approach. We have something that describes all the details of the requirements, but we haven’t considered implementation at all. Past experience tells us this may lead us into dangerous territory. Will the devil be in the details? Will the design have to change once we start implementing the abstractions? The first few times I did this, I was unsure. I was not even sure it could actually be made to work. The reason it does work is because of the way we have handled details. Firstly all details from requirements are in the diagram. The diagram is not an overview of the structure. It is the actual application. All other details, implementation details, are inside abstractions, where they are hidden even at design-time. Being inside abstractions isolates them from affecting anything else. So, it should now be a simple matter of writing classes for those three abstractions and the whole thing will come to life. Implementing the three abstractions turns out to be straightforward.

First, design some methods for the IConsistOf interface that we think we will need to make the system work:

    public interface IConsistsOf
    {
        void Ball(int score);
        bool IsComplete();
        int GetScore();
        int GetnPlays();
        IConsistsOf GetCopy(int frameNumber);
        List<IConsistsOf> GetSubFrames();
    }

The first four methods are fairly obvious. The Ball method receives the score on a play. The Complete, GetScore and GetnPlays methods return the state of the sub-part of the game. The GetCopy method asks the object to return a copy of itself (prototype pattern). When a child frame completes, we will call this to get another one. The GetSubFrames method is there to allow getting the scores from all the individual parts of the game as required.

The SinglePlay and Bonus abstractions are very straightforward.

So let’s code the interesting parts of the Frame abstraction. First Frame accepts a downsteam object that implements IConsistsOf. The WireTo method will set this field:

// Frame.cs
private IConsistsOf downstream;

Frame has one 'state' variable which is the list of subframes. This is the composite pattern we referred to earlier, and what ends up forming the tree.

// Frame.cs

private List<IConsistsOf> subFrames;
private readonly Func<int, int, int, bool> isFrameComplete;
private readonly int frameNumber = 0;

The second variable is the lambda expression that is passed to us by the application. It would be readonly (immutable) except that I wanted to use a setter to pass it in, not the constructor, to indicate optional configuration.

The third variable is the frameNumber, also immutable. It allows frame objects to know which child they are to their parent - e.g. 1st frame, 2nd frame etc. This value is passed to the lambda expression in case it is needed. For example, the lambda expression for a bowling frame needs to know if it is the last frame.

The methods of the IConsistsOf interface are now straightforward to write. Lets go over a few of them to get the idea. Here is the Ball method implementation for Frame:

public void Ball(int player, int score)
{
    // 1. Check if our frame is complete, and do nothing
    // 2. See if our last subframe is complete, if so, start a new subframe
    // 3. Pass the ball score to all subframes

    if (IsComplete()) return;

    if (subFrames.Count==0 || subFrames.Last().IsComplete())
    {
        subFrames.Add(downstream.GetCopy(subFrames.Count));
    }

    foreach (IConsistsOf s in subFrames)
    {
        s.Ball(player, score);
    }
}

It looks to see if the last child frame has completed, and if so starts a new child frame. Then it just passes on the ball score to all the child objects. Any that have completed will ignore it.

The IsComplete method checks two things: 1) that the last child object is complete and 2) that the lambda expression says we are complete:

private bool IsComplete()
{
    if (subFrames.Count == 0) return false; // no plays yet
    return (subFrames.Last().IsComplete()) &&
        (isLambdaComplete == null ||
         isLambdaComplete(frameNumber, GetnPlays(), GetScore()));
}

GetScore simply gets the sum of the scores of all the child objects:

private int GetScore()
{
    return subFrames.Select(sf => sf.GetScore()).Sum();
}

The GetCopy method must make a copy of ourself. This is where the prototype pattern is used. This involves making a copy of our child as well. We will be given a new frameNumber by our parent.

IConsistsOf GetCopy(int frameNumber)
{
    var gf = new Frame(frameNumber);
    gf.objectName = this.objectName;
    gf.subFrames = new List<IConsistsOf>();
    gf.downstream = downstream.GetCopy(0);
    gf.isLambdaComplete = this.isLambdaComplete;
    return gf as IConsistsOf;
}

The few remaining methods of the IConsistOf interface are trivial. The implementation of IConsistsOf for the other two abstractions, SinglePlay and Bonuses, is similarly straightforward. Note that whereas Frame uses the composite pattern, Bonuses uses the decorator pattern. It implements and requires the IConsistsOf interface. The SinglePlay abstraction, being a leaf abstraction, only implements the IConsistsOf interface.

One method we haven’t discussed is the wireTo method that we used extensively in the application code to wire together instances of our domain abstractions. The wireTo method for Frame is shown below:

public Frame WireTo(IConsistsOf c)
{
    downstream = c;
    return this;
}

This method does not need to be implemented in every domain abstraction. I use an extension method for WireTo. The WireTo extension method uses reflection to find the local variable to assign to.

The WireTo method will turn out to be useful in many ALA designs. Remember in ALA we "express requirements by composing instances of abstractions". If the 'instances' of 'abstractions' are implemented as 'objects' of 'classes', then we will use the wireTo method. If the 'instances' of 'abstractions' are 'invocations' of 'functions', as we did in the example project in Chapter One, we wont use WireTo obviously. In the coffeemaker example to come, 'instances' of 'abstractions' are 'references' to 'modules' because a given application would only have one of each abstraction.

The wireTo method returns 'this', which is what allows the fluent coding style used in the application code. The configuration setter methods also return the this reference so that they too can be used in the fluent style.

Here is the full code for the Frame abstraction (with comments removed as we just explained everything above):

// Frame.c
using System;
using System.Collections.Generic;
using System.Linq;
using GameScoring.ProgrammingParadigms;
using System.Text;

namespace GameScoring.DomainAbstractions
{

    public class Frame : IConsistsOf
    {
        private const int nPlayers = 2;         // TBD make this configurable
        private Func<int, int, int[], bool> isLambdaComplete;
        private readonly int frameNumber = 0;
        private IConsistsOf downstream;
        private string objectName;
        private List<IConsistsOf> subFrames = new List<IConsistsOf>();


        public Frame(string name)
        {
            objectName = name;
        }




        public Frame(int frameNumber)
        {
            this.frameNumber = frameNumber;
        }



        // Configuration setters follow.

        public Frame setIsFrameCompleteLambda(Func<int, int, int[], bool> lambda)
        {
            isLambdaComplete = lambda;
            return this;
        }





        // Methods to implement the IConsistsOf interface follow


        public void Ball(int player, int score)
        {
            if (IsComplete()) return;

            if (subFrames.Count==0 || subFrames.Last().IsComplete())
            {
                subFrames.Add(downstream.GetCopy(subFrames.Count));
            }

            foreach (IConsistsOf s in subFrames)
            {
                s.Ball(player, score);
            }
        }




        public bool IsComplete()
        {
            if (subFrames.Count == 0) return false;
            return (subFrames.Last().IsComplete()) &&
                (isLambdaComplete == null ||
                 isLambdaComplete(frameNumber, GetnPlays(), GetScore()));
        }




        public int GetnPlays()
        {
            return subFrames.Count();
        }




        public int[] GetScore()
        {
            return subFrames.Select(sf => sf.GetScore()).Sum();
        }



        List<IConsistsOf> IConsistsOf.GetSubFrames()
        {
            return subFrames;
        }




        IConsistsOf IConsistsOf.GetCopy(int frameNumber)
        {
            var gf = new Frame(frameNumber);
            gf.objectName = this.objectName;
            gf.subFrames = new List<IConsistsOf>();
            gf.downstream = downstream.GetCopy(0);
            gf.isLambdaComplete = this.isLambdaComplete;
            return gf as IConsistsOf;
        }

    }
}

The full source code for the bowling application can be viewed or downloaded from here: GameScoring code

2.12. Example project - Tennis

Now let’s modify the bowling application to score tennis. If the bowling game hadn’t been implemented using ALA, you probably wouldn’t contemplate doing this. But ALA excels for maintainability, and I want to show that off by changing Bowling to Tennis. The Frame and ConsistsOf abstractions look like they could be pretty handy for Tennis. A match consists of sets, which consists of games, which consists of SinglePlays.

We will need to make a small generalization to the Frame abstraction first. This will allow it to keep score for two players. We just change the type of the score from int to int[]. The Ball method will be generalized to take a player parameter to indicate which player won a play. A generalization of an abstraction to make it more reusable is a common operation in ALA.

The only other thing we will need to do is invent a new abstraction to convert a score such as 6,4 into a score like 1,0, because, for example, the winner of a game takes one point into the set score. This new abstraction is called WinnerTakesPoint (WTP in the diagram).

Here is the tennis scoring game:

tennis1.png

The diagram expresses all the details of the requirements of tennis except the tiebreak.

Here is the diagram’s corresponding code:

private IConsistsOf match = new Frame()
    .setIsFrameCompleteLambda((matchNumber, nSets, score) => score.Max()==3)
    .WireTo(new WinnerTakesPoint()
        .WireTo(new Frame()
            .setIsFrameCompleteLambda((setNumber, nGames, score) => score.Max()>=6 && Math.Abs(score[0]-score[1])>=2)
            .WireTo(new WinnerTakesPoint()
                .WireTo(new Frame()
                    .setIsFrameCompleteLambda((gameNumber, nBalls, score) => score.Max()>=4 && Math.Abs(score[0]-score[1])>=2)
                    .WireTo(new SinglePlay()))))));

The new WinnerTakesPoint abstraction is easy to write. It is a decorator that implements and requires the IConsistsOf interface. Most methods pass through except the GetScore, which returns 0,0 until the down-stream object completes, then it returns either 1,0 or 0,1 depending on which player has the higher score.

And just like that, the tennis application will now execute.

2.12.1. Add tiebreak

Now let’s switch our attention back to another example of maintenance. Lets add the tiebreak feature. Another instance of Frame will score the tiebreak quite nicely. However we will need an abstraction that can switch us from playing the set to the tie break. Let’s call it Switch, and give it a lambda function to configure it with when to switch to the tiebreak. Switch returns the sum of scores of its two downstream objects. Here then is the full description of the requirements of the tennis scorer:

tennis2.png

And here is the code version of that diagram. This application passes an exhaustive set of tests for the scoring of tennis.

private IConsistsOf match = new Frame("match")
    .setIsFrameCompleteLambda((matchNumber, nSets, score) => score.Max()==3)
    .WireTo(new WinnerTakesPoint("winnerOfSet")
        .WireTo(new Switch("switch")
            .setSwitchLambda((setNumber, nGames, score) => (setNumber<4 && score[0]==6 && score[1]==6))
            .WireTo(new Frame("set")
                .setIsFrameCompleteLambda((setNumber, nGames, score) => score.Max()>=6 && Math.Abs(score[0]-score[1])>=2)
                .WireTo(new WinnerTakesPoint("winnerOfGame")
                    .WireTo(new Frame("game")
                        .setIsFrameCompleteLambda((gameNumber, nBalls, score) => score.Max()>=4 && Math.Abs(score[0]-score[1])>=2)
                        .WireTo(new SinglePlay("singlePlayGame"))
                    )
                )
            )
            .WireTo(new WinnerTakesPoint("winnerOfTieBreak")
                .WireTo(new Frame("tiebreak")
                    .setIsFrameCompleteLambda((setNumber, nBalls, score) => score.Max()==7)
                    .WireTo(new SinglePlay("singlePlayTiebreak"))
            )
        )
    )
);

And just like that we have a full featured tennis scoring engine.

2.12.2. Final notes

Notice that I have added string names to the instances of Frame and other objects. This is not required to make the program function, but generally is a good habit to get into in ALA. It is because in ALA we typically use multiple instances of abstractions in different parts of the program. The names give us a way of identifying the different instances during any debugging. Using them I can Console.Writeline debugging information depending on the object’s name.

So there you have it. Around 8 lines of code express the scoring requirements of ten-pin bowling and around 15 lines of code express the scoring requirements of tennis. That sounds about right for the inherent complexity of the two games. The two scorers actually execute and pass a large battery of tests.

The domain abstractions are zero-coupled with one another, and are each straightforward to write by just implementing the methods of the IConsistOf interface according to what the abstraction does. The abstractions are simple and stable. So no part of the program is more complex than its own local part.

The domain abstractions are reusable in the domain of game scoring. And, my experience was that as the details inside the abstractions were implemented, the application design didn’t have to change. Here is a link to the code on Github: GameScoring code

Why two example applications? The reason for doing two applications in this example is to emphasis where all the details of the requirements end up. The only difference between the bowling and tennis applications is the two diagrams, which are translated into two code files: bowling.cs and tennis.cs of 8 lines and 15 lines respectively. These two files completely express the detailed requirements of their respective games. No other source files have any knowledge of these specific games. Furthermore, Bowling.cs and Tennis.cs do not do anything other than express requirements. All implementation to actually make it execute is hidden in domain abstractions and programming paradigm abstractions.

3. Chapter three - Why the structure works

In the previous chapter we described what the structure, the anatomy, of ALA looks like as if we were dissecting a dead body. We see where things are but we don’t yet understand why they are there. In this chapter we focus on explaining why that structure works. Why does this way of organising code result in software that meets those non-functional requirements we listed in Chapter one?

3.1. Good versus bad dependencies

We can distinguish two types of dependencies. On one hand there are run-time dependencies. These are dependencies in the code that are there because one module will need another module to be present at run-time for the system to work. On the other hand there are design-time dependencies. These are things that you need to know about to even understand a given piece of code. I will often refer to this type as a "knowledge dependency". It is also sometimes called "semantic coupling".

Run-time dependencies are bad.

Design-time dependencies are good.

A simple example of a run-time dependency is a module that calculates the average rainfall calling a module to display the result. The Display module needs to be present at run-time. But to understand the code that calculates the average rainfall requires no knowledge about displays, nor even where the result will be sent. The dependency is only there to meet a run-time requirement.

A simple example of a design-time dependency is some code that calculates the standard deviation. To understand the code needs knowledge of squareroot. This is a design-time, knowledge dependency. Any standard deviation code should use and have a dependency on squareroot.

We find both types of dependencies in conventional code. Whether a knowledge dependency or a run-time dependency, they all just look like a function call or a 'new' keyword. We generally don’t distinguish between them. They are all just called dependencies. We lump them together when we talk about dependency management, loose coupling, layering, or circular dependencies. Dependency graphing tools show them both, and try to turn them into a high number of 'levels' (if circular dependencies are not present.)

It turns out that a knowledge dependency is good if it is on a module that is even more abstract than itself. That’s what gives you the power of compositionality - to compose new but less abstract creations from more abstract building blocks. The more knowledge dependencies, the more you are using those reusable abstract building blocks. The more the better. More is better because if something is used a lot, it follows that it is a good abstraction. And if something is a good abstraction, it is stable.

Run-time dependencies are always bad. They are bad because they always destroy abstractions. And they are bad because they obscure the structure.

In ALA we eliminate all run-time dependencies.

That’s right - in ALA we eliminate all the run-time dependencies!

Consider the diagram below. There are four explicit run-time dependencies.

dependency diagram.png

A typical program is chock full of these types of dependencies.

These dependencies have the following consequences:

  • Function names in A, B, D and E will likely relate to what they do. This may include the name of the specific hardware chips used in the drivers. So the celling code must know these names, tying it to these specific modules.

  • Since there is now a fixed arrangement from C to the others, it is likely to be collaborating with them, meaning that C is making implicit assumptions about some details of what B and D do, and similarly for the others.

  • Although there is no explicit dependency from A to B or B to C, etc, the fixed arrangement is likely, over time, to make A collaborate with B (do what B requires), B collaborate with C, etc effectively making implicit dependencies.

  • The collaborative arrangement between A, B, C, D and E is not obvious from the outside. It is buried inside of B, C and D, and to some extent inside A and E as well. (There is no diagram like the one above to tell us this arrangement, or if there is, it is a second source of truth and is likely out of date and lacking in details.)

  • The fixed arrangement makes it impossible to reuse C without A, B, D and E. Only A and E can potentially be abstractions.

  • Arbitrarily, only the two ends of the data flow chain can be reused.

  • The fixed arrangement makes it difficult to insert another operator between A and B or between B and C, etc.

  • If the observer pattern is used, so that the compile-time dependencies are reversed, it only mirrors the same problems. And because it adds indirection, the observer pattern makes the program even harder to understand.

  • If dependency injection is used with automatic wiring, the fixed arrangement may be somewhat loosened. For example A may be any class that implements the same interface. But many of the implicit dependencies can remain. All classes can still be collaborating with one another to some extent. A smell that this is happening is that over time the interfaces of A, B, D and E change as the requirements of the system change. Worse, the implicit collaboration between A, B, C, D and E is even more obscure - now we have the worst of both worlds.

During code creation, run-time dependencies are easily introduced, and never seem too terrible at the time. But when they accumulate to hundreds or even thousands of them, that’s when the whole system becomes a monolith.

So how do you eliminate all run-time dependencies? And how do you eliminate the implicit collaboration? While we will use dependency injection, observer (publish/subscribe) pattern, callbacks, or the like, they don’t solve the problem by themselves. Something more strategic is required.

To completely remove the run-time dependencies, the five modules must know nothing about each other. So not loosely coupled. Zero coupled. They each do a job using their input and output ports, and know absolutely nothing of each other. When you do that, we call the module an abstraction. Something else, let’s call it the 'application' knows that in a particular case, A, B, C, D and E will talk to each other at run-time, or more accurately, instances of A, B, C, D and E will talk to each other. That wiring is explicit, and in one place. It is not obscured inside of A, B, C, D and E.

dependency diagram 1.png

The application has knowledge dependencies (design-time dependencies) on A, B, C, D and E. The letters used in the top layer represent instances. In common programming languages these instances are in the form of either new A() or just a function call, A(). These are good, because the application knows it needs to compose these abstractions in a particular way to make a particular system. The run-time dependencies that used to be directly between A, B, C, D and E, are gone. This application level module either moves the data between the instances of A, B, C, D, E itself, or wires them together using even more abstract interfaces. These interfaces are not specific to any of A, B, C, D or E. This is the abstract interactions pattern. The interface design is such that there could potentially be many abstractions that either implement it or accept it. Then the application has choices about how it combines abstractions. This is called compositionality.

When using the observer pattern, the application, does the subscribing for the receiver. The receiver must not do the subscribing itself for it shouldn’t know where it’s data come from.

When using dependency injection, the wiring must not be automatic. It must not just grab objects from a container that matches interfaces. This would make the wiring implicit and the application even more difficult to understand. It will also encourage specific interfaces for pairs of classes, which is bad. I prefer not to use XML for wiring either. XML is not very readable, and it only handles tree structures well. If you must use text, use normal code. But there are situations where a diagram is the only readable way to go. I will go into these in a later section.

When you structure code in this way, all the run-time dependencies have gone. You may be wondering, where did they go? How can the program work without them? Or haven’t I just moved them somewhere else? No. As dependencies, they are gone. The knowledge that A will talk to B, B to C and so on is still there, just not as dependencies. That knowledge is now in ordinary code, fully contained in one place now.

The only dependencies that remain are knowledge dependencies:

  1. Dependencies from the application to the domain abstractions. The application 'knows' at design-time what abstractions it needs instances of.

  2. Dependencies from the domain abstractions to the abstract interfaces. Domain abstractions know at design-time what types of interfaces they will need to input or output.

3.2. Stability layers

The elimination of all run-time dependencies that we discussed in the previous section changes the nature of layering. The layers now only involve knowledge dependencies (design-time dependencies). Lower layers provide the building blocks (the abstractions) for higher layers. These layers have different levels of abstraction with each lower layer being a big step more ubiquitous, more reusable and more stable. This results in only a few layers.

To understand code, you have to learn and know all the abstractions available to you in lower layers.

The bottom layer is your general purpose programming language. You must know its abstractions such as functions and if-else statements before you can understand any layer above. You can generally learn this once for a whole career.

You also need to know the next layer, which is at the abstraction level of programming paradigms. Examples are state machines, data-flows, database schemas, and UI trees. You would generally learn these as needed for different programming problems. A given domain will typically use several of them.

You also need to know the next layer, the domain layer where you have useful building block for solving problems in a specific domain. You would learn these when you start a new job.

Finally we come to the top layer. Its abstraction level is a single application. The abstraction is what the user sees - a tool that does a job by meeting a set of requirements. The abstraction level of the top layer is the details of the requirements.

To get the insight of ALA, you need to throw away any previous conceptions of layering you may have had as these will contain run-time dependencies. Think of run-time dependencies as just wiring in the top layer, tipped on its side. ALA’s layers mean that all dependencies are toward more stable abstractions.

One of the guidelines sometimes used for dependencies is that a class that has high fan-in should not have high fan-out. The argument goes that a class with high fan-in should have high stability and one with high fan-out would have low stability. In ALA we modify this idea into stability layers. Each layer is significantly more stable than the one above. We can obviously only achieve this with a small number of layers. But a class in the middle layer can indeed have both high fan-in and high fan-out. Every class obviously has high fan-out on the bottom layer, the general purpose programming language. You definitely want your programming language to be stable.

3.3. Expression of requirements

We have previously discussed this aspect of ALA in terms of structure. It is the top layer. And we have used this aspect as the starting point in the example projects. But why does the succinct description of requirements in that top layer work?

In conventional software development, we typically break a user story (or feature or functional requirement) up into different implementation responsibilities. For example, 'layers' like GUI, business logic and database, or a pattern such as MVC (Model, View, Controller). But a user story or feature actually starts out as cohesive knowledge. And its not a huge amount of cohesive knowledge, so it doesn’t need breaking up. Cohesive knowledge, knowledge that is by its nature highly coupled should always be kept together. All we need to do to keep it together is find a way to describe it. Don’t try to do any implementation of, just get it described in a precise and complete form. If you can do that, the chances are you will be able to find a way to make it execute.

In ALA we want to find a way to express the user story with about the same level of expressiveness as when the user story was explained in English by the product owner. The language he used would have contained domain specific terms to enable him to explain it concisely. The same thing ought to be possible in the code. Any knowledge that does not come directly from the requirement specification is separated out. It comes out into separate abstractions. These abstractions typically contain knowledge of how things are implemented. But not how a specific user story is implemented - how different aspects of typical user stories are implemented.

It turns out that abstractions that know how to implement useful things for expressing user stories are not only reusable for different user stories, but can be reusable for other applications. In other words, they are domain level abstractions. A typical user story might be composed of several of them, some to implement the user story’s UI, some to implement the user story’s business, and some to implement the user story’s data. A user story instantiates the abstractions, configures them with the specific knowledge from the requirement, and then wires them together.

Generally, maintainability of ALA architectures for any kind of change is good because all separation of code is done by use of abstractions, which are naturally decoupled. For example, most maintenance is probably fixing bugs. But the next most common form of maintenance is probably adding features. Here is where the top layer code comes into its own. Because you just describe requirements using domain abstractions, adding new features gets faster and faster as the domain abstractions mature.

3.4. Abstractions are design-time encapsulators

The maintainability quality attribute is often thought of in terms of ripple effects of change. I don’t think that is quite the right way to look at it. I have often had to make changes across a number of modules in poorly written code. The changes themselves just don’t take that long. The problem I see is the time you have to spend understanding enough of the system to know where to make those changes and to be confident they wont break anything. Even if the change is one line of code (which it often is), the potential for breaking something depends on the complexity. You have to understand all the relationships and coupling to that one line of code.

Unlike modules, abstractions encapsulate complexity at design-time. They give boundaries to how far you have to read code to understand code.

3.5. Diagrams vs text

Why do we often use diagrams instead of text in the top layer of an ALA application?

Text can readily be used to compose abstractions in a linear chain using spaces. Sometimes periods or → or other symbols are used instead. Text can also handle tree structures using brackets, usually () or {}. The brackets work for the compiler, but the brain doesn’t see them, so we typically also use indenting. When the tree gets deep, the indenting is too deep for our brains to see. We can say, then, that text is only suitable for trees that are not deep. Structured programming and XAML are examples of tree structures represented successfully in text.

When we need to compose abstractions in an arbitrary graph structure, our brains work much better using a diagram. The text version uses uses 'label connections', which our brain can only see through an editor that highlights them (for small scopes), or an all-files search.

When we talk about connection through labelled references, we are talking about references that are not abstractions. References to the names of abstractions are fine and we don’t draw lines for them even if we are using a diagram. What we are talking about here are connections between elements that are just connections, not abstractions. In other words indirection without abstraction. It just doesn’t work except in very small scopes, for example a local parameter of a function. With a diagram the brain can readily see the lines, which are fine being anonymous. The spacial positioning of elements is also something the brain readily remenbers. So diagrams can do what text cannot. Imagine an electronics engineer designing a circuit without a schematic. If you turn a graph into a list of labelled connections, the brain cannot reason with it without first turning it back into a diagram. It’s the same in software. When you start describing your requirements by inventing and composing domain abstractions, they usually, have a network structure.

The design process of describing your requirements and inventing the needed abstractions as you go is an intense activity that produces the system architecture. When I am doing it, I have no brain power left over for any overheads. Text is out of the question. A diagramming tool that constantly gets in your way, such as Visio, is also out of the question. Doing diagrams directly on paper with a pencil is also too hard as you very quickly need to keep relaying it out.

Until there is a better tool, I have been using Xmind, the mind mapping tool. It seems to only need about 1% of brain power to drive it. I use some simple conventions to get around its limitations. It lays itself out using a tree structure but allows quick 'cross connections' to be added easily.

When we build a diagramming tool, it will have the low driving overhead of a mind mapping tool. As with a mind-mapping tool, it will use keypresses for the majority of the work, and allow mouse clicks when it makes sense e.g. to specify the destination of a 'cross connection'. Like a mind mapping tool, the layout will be automatic and will follow the 'primary tree' structure that you create. You can of course make 'cross connection' that don’t influence the layout, but which the tool will route for you neatly. You can make subgraphs that are connected to the rest of the tree for the purpose of automatic layout, but logically disconnected (just an invisible line). It can of course have 'cross connections' to it.

You can think of a conventional mind-mapping tool tree as having branches that expand out in one direction. This new tool will allow the branches from any node to go in any of the four directions. This makes sense if you think of the nodes as boxes and the branches as ports on those boxes. You can put your ports anywhere around the box. Shift arrow keys, you create a port and another box in the desired direction. This defines the layout, and the boxes will spread themselves out spacially automatically. Then you can add cross connections as desired.

Although I have been successful using Xmind, such a tool would aid the design phase. Of course the tool would also automatically generate the corresponding code.

3.6. Composability and Compositionality

We have referred to the property "composability" a few times. By composability, we refer to the ability to create an infinite variety of applications by combining a finite number of domain abstractions in different ways.

This is an important property in ALA. Once a reader know a finite number of abstractions together with their rules of composition, they are able to understand an potentially infinite number of compositions first reading.

Composability uses the Principle of Compositionality which states: In mathematics, semantics, and philosophy of language, the principle of compositionality is the principle that the meaning of a complex expression is determined by the meanings of its constituent expressions and the rules used to combine them.

In software engineering, it is described by a pattern called "Abstract Interactions" or "Configurable Modularity" by Raoul de Campo and Nate Edwards - the ability to reuse independent components by changing their interconnections but not their internals. It is said that this characterises all successful reuse systems, and indeed all systems which can be described as "engineered".

ALA has these properties using domain abstractions and programming paradigm interfaces.

As mentioned earlier, there are other software systems that have composability, usually using the data-flow paradigm, such as RX (Reactive Extensions). Most composability systems are restricted to a single paradigm. For ALA to have the correct level of expressiveness when inventing and composing domain abstractions to describe requirements, a variety of 'connection paradigms' are needed. Some exmples of these are discussed in the next chapter on execution models.

We can make an analogy with Lego bricks. Some Lego parts have the familiar little stud and tube connectors. Some will support axles and holes connections, either tight or loose. These different ways of connecting Lego parts are analogous to different programming paradigms and different ways for the modle to 'execute' at run-time.

If the domain were for building model toys (the Lego domain), the non-ALA method would start with the imagined toy and decompose it into parts specific to that one toy. The solution would be brittle and hard to change and no other toys would be possible without the same huge effort all over again. The ALA method is to invent a finite set of building blocks and the mechanical paradigms by which they connect. Then the initial toy can be easily changed, and other toys are possible for little effort.

3.7. Example project - Some real dependency graphs

Like the previous example project, this is a legacy application that is approximately 10 years old and still under maintenance. Consequently it is a big ball of mud. Maintenance has become so difficult that it was decided to rewrite it using ALA as a research project. This gives us a good basis for comparison.

The original application has around 80 KLOC. Rather than look at any of the details of the application itself, we present here dependency graphs generated by Ndepend for the old legacy application and new ALA application.

3.7.1. Legacy application dependency graphs

One of the core tenets of ALA (as discussed in Section 3.2) is "Composition using layers" instead of "Decomposition using encapsulation". Unfortunately Ndepend is designed with the assumption that the application should be built using the latter approach. It likes to present a decomposition structure, starting with assemblies (packages) at the outermost level, then namespaces, and then classes. I’m not sure why it considers namespaces a viable mechanism because they don’t provide encapsulation. Anyway, here is the namespace dependency graph for the main assembly of the legacy version of the application, as it comes out of ndepend.

namespaces.png
Figure 7. Legacy application - namespaces

This graph is quite large, so if you like you can right click on it, and open it in a new tab in your browser to view it. The red arrows are dependencies in both directions.

Each box represents a namespace. The thickness of the arrows is proportional to the number fo dependencies. The size of the boxes is proportional to the number of lines of code in the namespace.

As is typical, namespaces provide no useful decomposition structure in this application. They do not make abstractions in themselves, nor do they implement a facade pattern or an aggregate root type of pattern with even logical encapsulation. Any classes inside each namespace have unconstrained relationships with any classes in any other namespace.

If we drill down into the largest namespace, UIForms, we see the class relationships between classes inside that namespace:

classes in uiforms namespace.png
Figure 8. Legacy application - classes in uiforms namespace

Here you can see that ndepend is trying to make out the layers. The layers are vertical columns, going from left to right. I have left them vertical even through ALA abstraction layers are usually drawn horizontal because they come out more readable on the page. Again there are many dependencies in both directions drawn in red.

Here are the classes inside the DataStructure namespace:

classes in datastructure namespace.png
Figure 9. Legacy application - classes in datastructure namespace

Again, Ndepend is trying to make out the layers from left to right.

There is one class called Device which actually looks like it might be a good abstraction.

Ndepend is giving us a false picture here, because it is omitting all dependencies that go in or out of the namespaces. To really get an idea of what the big ball of mud looks like, I configured Ndepend to use a query that gives me all the classes in all the namespaces. Here finally is what this application truly looks like.

classes in all namespaces.png
Figure 10. Legacy application - all classes in all namespaces

This graph is very large. Right click on it, and open it in a new tab in your browser, so you can zoom in to see the dependencies in the background. It is truly frightening. Ndepend had no chance to find the dependency layers. It makes visible, for the first time, why continued maintenance on this application is so difficult. You have to read a lot of code to find even a tiny part of this hidden structure.

The developer who maintains the application tells me this is a fair projection of the complexity that he has to deal with.

To be fair, some of the dependencies in this diagram are 'good' dependencies, dependencies on what might be abstractions if the rest of the dependencies were not there (Section 3.1 discusses good and bad dependencies). For example, the box near south-east called ScpProtocolManager has a lot of dependencies coming into it, which means it is possibly used a lot and therefore is a potential good abstraction.

3.7.2. New ALA application dependency graphs

Here is the equivalent ndepend generated class dependency graph for the new ALA version of the application.

classes in all namespaces.png
Figure 11. New ALA application - classes in all namespaces

Ndepend has tried to find the three ALA layers which are vertical and go from left to right. Only the Application sits in the top layer. The DomainAbstractions layer contains the next two columns of classes and a few from the next column. And the ProgrammingParadigms layer contains the rest on the right. Actually there were a couple of errors in the dependencies when this graph was gnerated which have since been fixed. (There should be no dependency between Panel and OptionBox, nor between Wizard and WizardItem.) As features are added, this is all the dependencies you will ever see in this applciations. The Application already uses all the domain abstractions, and the domain abstractions each use most of the programming paradigm interfaces, as they should. There are a few DomainAbstractions to be added, but this is essentially what an ALA class dependency graph across all layers looks like to ndepend. All the dependencies shown are good dependencies - design-time dependencies on abstractions. There are no run-time dependencies in the graph.

Just for interest, here is ndpend’s namespace dependency graph.

namespaces.png
Figure 12. New ALA application - namespaces

Remember in ALA, we do not use decomposition, so namespaces do not represent decomposition of the system. They represent layers. You can clearly see the three layers. The wiring namespace also goes in the programmingparadigms layer.

Lets drill inside the domain abstraction layer to see the interdependencies within that layer. We expect to see no dependencies.

classes in domainabstractions namespace.png
Figure 13. New ALA application - classes in DomainAbstractions namespace

Ok here we see the two previously mentioned dependencies that are in error and two other dependencies. They are on delegates or enums in the same source file, and so don’t really count as bad dependencies.

And finally, let’s drill into the ProgrammingParadigms namespace

classes in programmingparadigms namespace.png
Figure 14. New ALA application - Classes in Programming Paradigms namespace

Again we see a few dependencies on delegates which are ok. There is a couple of connector classes that depend on interfaces in this same layer. They definitely belong in this abstraction layer, and definitely aren’t run-time dependencies, so it looks like we don’t have a perfect score of zero dependencies within a layer.

As of this writing, the new ALA version of the application is still a research project, but so far everything has gone smoothly with two weeks spent doing the description of the requirements as a diagram, potentially executable, and three months so far spent writing the domain abstractions. So far there are no great issues getting it to actually execute. It is expected that it will be commercialized soon and replace the old application.

4. Chapter four - Execution models

4.1. Execution models (Programming paradigms)

ALA requires the use of execution models other than the native CPU’s hardware sequential instruction execution flow (called imperative). The imperative paradigm of the hardware stays as the major paradigm in most general purpose high level languages. Some programmers are so used to thinking in terms of sequential execution of statements that it can be very difficult to think in terms of anything else. You keep wanting to know what the equivalent sequentially executed model is in order to understand what is going on instead of letting go and just thinking in terms of the different programming paradigm. Certainly it is nice to know what is going on under the covers from a performance or resourcing point of view. But from a logical point of view, it is better to let go and start thinking in a whole range of different ways.

State machine execution model
Figure 15. State machine execution model

The canonical example is a state machine. At first it can be difficult to write a program as a state machine, even if a state machine is a more suitable execution model. It takes some getting used to.

It is an essential ingredient of ALA to make use of multiple execution models. This is sometimes referred to as polyglot programming paradigms. For most computing problems, the most common paradigm needed is not imperative but a combination of data-flow, UI layout, data schema and activity.

Following is a selection of common programming paradigms with a short description of each. This list will give you an idea of how varied they can be, and powerful using them in combination would be. It is not an exhaustive list. In ALA you can invent your own programming paradigms if they better express the requirements, just as we did with the 'IConsistsOf' model that we used in the game scoring domain in the project example in the previous chapter.

4.2. Imperative

The default provided by your programming language. It reflects the underlying machine. Connected element are executed consecutively and synchronously as fast as the CPU can do them. The connected elements are language statements, especially function or method calls. Functions and methods are executed 'synchronously', which means that execution is always passed with the messages. The receiver of a message always gets both the message and the CPU resource to process it. When done, the receiver always returns the CPU recorce to the sender where it resumes execution. This paradigm is only suitable when you know ahead of time the order that things will happen, and those things should happen as fast possible with no temporal concerns whatsoever. It is the paradigm to use to execute short running algorithms.

4.3. Event driven

'Event driven' is an overloaded term in software engineering because it generally means both 'an asynchronous message' and 'decoupled'. Let’s clarify the decoupled part briefly because that part doesn’t involve the execution model.

4.3.1. coupled/decoupled

One perspective of 'Event Driven' is simply breaking out of the imperative 'synchronous/coupled' paradigm. In Imperative programming, function calls are synchronous because the caller waits for it to return before continuing its own execution. Function calls (between modules) are coupled because the function caller refers directly to a function in another module by its name.

'Event driven' usually means both 'asynchronous' and 'decoupled'. It is asynchronous because the sender of the message does not wait for a response - it continues with its own execution immediately. It is decoupled because the sender does not name the recipient of the message. Note that while that is considered decoupled in Event Driven programming, it is not considered decoupled in ALA because the coupling is simply reversed - the receiver of the message names the sender (or the event name). In ALA neither coupling is allowed.

In the real world we don’t normally think about asynchronous or decoupled - we just think of them as events. When an event happens, it doesn’t wait for everyone interested in it to finish being interested before moving on, so it is asynchronous. And the entity that causes the event doesn’t know who is interested, so it is decoupled (in that direction).

These two notions, synchronous/asynchronous and coupled/decoupled are actually independent of each other. All four combinations are possible and sensible. In fact all four combinations are used in ALA. Although the term 'Event Driven' usually refers to the combination asynchronous/decoupled, in C#, events are synchronous/decoupled, being a synchronous implementation of the observer pattern. The coupling/decoupling aspect has nothing to do with execution models. We have already established ALA’s rules on coupling in chapter three - you must be decoupled (in both directions) unless using an abstraction in a lower layer.

So now lets just look at the Event Driven programming paradigm from the perspective of being asynchronous.

4.3.2. synchronous/asynchronous

Events are things that can potentially happen at any time, so we lose the Imperative notion of knowing ahead of time what the CPU will be doing in what precise order (not withsatnadning that in "Event Driven" there can be planned sequences of events, for example during request/response I/O). Event Driven can thought of as a 'reactive' execution mode. Events fire and short routines run in reaction to them. We also lose the notion of execution in CPU time. The CPU resource is not passed with the event as it is in Imperative. Function calls that signal events always return immediately (whether coupled or decoupled). A scheduler of some sort must figure out how the CPU will get to execute all the tasks that are ready to react to those events.

ALA can use both asynchronous and synchronous. It does not have rules for when to use one or the other. The rules remain more or less the same as in non-ALA applications.

Asynchronous is the more general case. However there are reasons to prefer synchronous. Synchronous is more efficient because it is directly supported by the silicon. Synchronous has less potential problems because it inherently orders the execution of everything whereas asynchronous leaves the order of execution to be defined separately. Synchronous allows request/response to be coded with a simple function call, which is nice (although languages that have async/await can do it with the asynchronous execution model just as nicely). Synchronous is of course only possible locally and on the same thread. Synchronous requires the receiver of the message to process it immediately, quickly, and without blocking. So synchronous cannot be used in situations when the receiver of an event is not local, there is slow IO involved, or the processing will take a long time.

ALA is all about abstractions that know nothing about each other. It is desirable that abstractions that generate events and abstractions that listen to events don’t have to be coupled by having to know whether they will both use synchronous or both use asynchronous. The only way around this problem is for any interfaces that may need to be either to be asynchronous. That’s because aynchronousk is the more general case. An asynchronous interface can be used synchronously.

Event listeners implementing an asynchronous interface can choose to handle the event synchronously if they wish. If callbacks are used, this means that they will call the callback function in the interface synchronously. If Tasks or Promises are used, it means they will return a Task or Promise already in the complete state.

Event generators implementing an asynchronous interface must inherently handle asynchronous. If callbacks are used, it needn’t care if the callback is called back synchronously when it sends the event. If Tasks or Promises are used, it needn’t care if the Task or Promise it gets back when it sends the event is already in the complete state.

Unfortunately, if you make an interface asynchronous in order for it to be able to handle either the asynchronous or synchronous execution model, both ends must be written in the style of asynchronous. Unless using async/await, the asynchronous style can be very awkward in some scenarios. This is especially true when there is a known sequence of events to be done that is naturally expressed as sequencialy function calls. The other problem is that the async functions can start spreading to everywhere.

This affects code that is written inside domain abstractions. It doesn’t affect things so much at the application layer. This is because in the application layer we have lifted ourself into the realm of composition of instances of abstractions. If you are wanting to do something sequential (not Imperative) in the application layer module, you do it using the 'Activity' execution model which we will describe in the next section.

4.3.3. Preemptive/non-premptive

Before leaving the 'Event Driven' execution model, we just need to clarify two variants of asynchronous - preemptive or non-preemptive. An Event or Message can be sent and end up being executed on another thread or the same thread.

In ALA, when using asynchronous, it must be non-pre-emptive by default. This is to prevent thread safety causing coupling between abstractions. Preemtive asynchronous (using multiple threads) would only be used when it is the only way to solve temporal constraints, which would be understood to be compromising the decoupling between abstractions. This is the same criteria you should use in any type of programming style.

4.3.4. Examples of Event Driven

There are many examples of usage of event driven such as in Node.js or the reactor pattern, and there is usually an Asynchronous Event Framework behind the scenes of state machines.

4.4. Activity-flow

As for a UML activity diagram. Connected activities start when the previous one finishes. The activity may take a long time to complete without holding the CPU. Activity flows can split, run in parallel or pseudo-parallel and recombine. They can be asynchronous.

There are languages or libraries that support the Activity paradigm in text so that the code looks like the Imperative paradigm but is actually more of the Activity Paradigm. These are mechanisms such as async/await, yield, or coroutines such as Protothreads implemented using Duff’s device in C.

4.5. Work-flow

Persisted Activity-flow. This includes long running activities within a business process such as an insurance claim.

4.6. Data-flow

A data-flow model is a model in which adjacent instances in the program (or connected boxes on a diagram) are a path of data instead of a path of execution. The execution flow is like in another dimension relative to the data flow - all over the place. The ALA architect will even invent new execution models, whatever makes the expression of those requirements convenient and declarative.

A stream of data flows between the connected components. Each component processes data at its inputs and sends it out of its outputs.

Each input and output can be operated in push or pull mode. Usually the system prescribes all pull (LINQ), all push (RX), inputs pull and outputs push (active objects with queues) or outputs pull and inputs push (active connectors).

The network can be circular provided some kind of execution semantic finishes the underlying CPU execution at some point, such as using ticks to move data between components as done in function blocks.

The data-flow paradigm raises the question of type compatibility and type safety. Ideally the types used by the components are either parameterised or determined through type inference.

4.7. Live-data-flow

As used in the coffee-maker example earlier, this paradigm simulates electronic circuits. Semantically the outputs and inputs are live at all times. This type of flow is most readily implemented with shared memory and on-change events.

4.8. State machine (transition-flow)

4.9. UI layout

4.10. UI navigation flow

4.11. Data schema

4.12. Example Project - Coffee machine

Robert Martin posed an interesting pedagogical sized embedded system problem about a coffee maker in his book “Agile Software Development: Principles, Patterns and Practices”. The original chapter is called “Heuristics and Coffee”.

In the original chapter, the worked solution to this problem uses decomposition into three modules that collaborate or interact with one another. The ALA solution follows the opposite philosophy. It has three abstractions (which correspond with the three modules), but they do not collaborate or interact with one another. Being abstractions, they don’t know anything about each other. As domain abstraction, they know nothing about the coffee machine. The coffee machine is then constructed as another abstraction, in the top layer, that makes use of the three domain abstractions.

This example uses different execution models from the 'consistsof' and 'dataflow' ones that were used in the two game scoring examples of previous chapters. Here we will use some interesting electronic-signal-like execution models that have an extremely simple main-loop polling type implementation, just as Robert Martin’s original solution also had.

Reading an ALA application requires first paying attention to the pre-requisite knowledge you need from lower layers. So before presenting the application, lets first familiarise ourselves with the abstractions we need from the domain layer, and the Programming Paradigms layer.

4.12.1. Domain abstractions layer

Here are the three domain abstractions:

CoffeeMaker Domain Abstractions
Figure 16. Coffee maker domain abstractions

Take a moment to look at these three abstractions:

 — The UI has a lamp you can control, and a push button which outputs an event.

 — The WarmerPlate tells you whether or not the pot is on the warmer plate, and whether or not it is empty. It controls its own heater.

 — The Boiler can be turned on or off. It will tell you when it is empty of water. And you can stop water flow instantly with a steam release valve. It will turn its own heater off if it runs out of water, or the valve is opened.

That’s all there is to know about the three domain abstractions.

4.12.2. Programming Paradigms layer

We have three programming paradigms

 — live-data-flow (works like an electronic circuit)

 — events

 — simple state machine

The API for the Programming Paradigms layer is described in the key on the right of the diagram below. It gives you all the knowledge from this layer to be able to read the diagram. So, for example, a solid line is a data-flow; the rounded box is state with the states enumerated inside it.

The details of how to turn the diagram into code is explained in a project document, also provided in the Programming Paradigms layer.

4.12.3. Application layer

Now that we have understood the knowledge dependencies in all lower layers, we can read the diagram that resides in the top layer, the application layer.

CoffeeMaker Dataflow diagram
Figure 17. Coffee maker solution

The diagram to the left is the application itself. Instances of the three domain abstractions, UI, Boiler and Warmer plate are shown as boxes.

Follow me now as we go through the user stories by looking at the lines on the diagram:

  • When the UI push button is pressed, we set the state to Brewing, provided the Boiler has water and the pot is on the Warmerplate.

  • When the state is brewing, it turns on the boiler, and coffee making starts.

  • If someone takes the pot off, the valve is opened to momentarily release pressure from the boiling water, which stops the water flow.

  • When the boiler becomes empty, the state is set to Brewed. When the state is Brewed, the light in the UI is turned on.

  • When the coffee pot is replaced empty, the state goes back to the idle state where we began.

That’s all there is to reading this application. The code for the coffee machine can be read and understood in about one minute. Compare that with reading other solutions to the coffee machine problem.

Note that the paragraph above is pretty much a restatement of the requirements in English. It could have been the requirements. The amount of information in the English form (or the diagram form) is about the same, thus the Domain Abstractions gave us the correct level of expressiveness. Further confirmation of this is if the level of expressiveness allows us to modify it.

For example, say a requirement was added that a coin device was to enable the machine to be used. The coin device is an abstraction that provides an output when a coin is given, and has a reset input. Looking at the diagram, and being able to reason about its operation so easily, you can see that the coin device’s output would intercept the Pushbutton using another instance of an AND gate. And to reset the coin device, you could use the boiler empty output event.

4.12.4. Execution

To make it actually execute, we apply the manual procedure documented in “Execution models.doc”. This document is in the Programming Paradigms layer. It will generate these 6 lines of code:

if (userInterface.Button && warmerPlate.PotOnPlate && !boiler.Empty) { state = Brewing; } userInterface.Button = false;
boiler.OpenSteamReleaseValve = !warmerPlate.PotOnPlate;
boiler.On = state==Brewing;
if (boiler.Empty && !prevBoilerEmpty) { state = Brewed; } prevBoilerEmpty = boiler.Empty;
if (warmerPlate.PotEmpty && !prevPotEmpty) { state = Idle; } prevPotEmpty = warmerPlate.PotEmpty;
userInterface.LightOn = state==Brewed;

There is a one-to-one correspondence between the lines in the diagram and the code.

As you can see, the execution model is a simple one. The 6 lines of code are continually executed. This execution model is effective and appropriate for this small application.

The 6 lines of code can be built into a complete program shown below:

 #ifndef _COFFEE_MAKER_H_
 #define _COFFEE_MAKER_H_
 // Coffee Maker domain abstraction
 #include "CoffeeMakerAPI.h"  // original hardware abstraction supplied by hardware engineers
 // Knowledge dependencies :
 // "PolledDataFlowProgrammingParadigm.doc" -- explains how to hand compile a data flow diagram of this type to C code
 // Following are 3 Domain abstractions that the application has knowledge dependencies on



 #include "UserInterface.h"
 #include "Boiler.h"
 #include "WarmerPlate.h"



 class CoffeeMaker
 {
 private:
    enum {Idle, Brewing, Brewed} state;
    Boiler boiler;
    UserInterface userInterface;
    WarmerPlate warmerPlate;
    bool prevBoilerEmpty, prevPotEmpty;
    void _Poll();
 public:
    CoffeeMaker()
        : state(Idle), prevBoilerEmpty(boiler.Empty), prevPotEmpty(warmerPlate.PotEmpty)
    {}
    void Poll();
 };
 #endif //_COFFEE_MAKER_H_
 // CoffeeMaker.c
 // This is not source code, it is code hand compiled from the CoffeeMaker application diagram
 #include "CoffeeMaker.h"

 void CoffeeMaker::_Poll() (1)
 {
    if (userInterface.Button && warmerPlate.PotOnPlate && !boiler.Empty) { state = Brewing; } userInterface.Button = false;
    boiler.OpenSteamReleaseValve = !warmerPlate.PotOnPlate;
    boiler.On = state==Brewing;
    if (boiler.Empty && !prevBoilerEmpty) { state = Brewed; } prevBoilerEmpty = boiler.Empty;
    if (warmerPlate.PotEmpty && !prevPotEmpty) { state = Idle; } prevPotEmpty = warmerPlate.PotEmpty;
    userInterface.LightOn = state==Brewed;
 }



 void CoffeeMaker::Poll()
 {
    // get inputs processed
    userInterface.Poll();
    boiler.Poll();
    warmerPlate.Poll();
    // run application
    _Poll();
    // get outputs processed
    userInterface.Poll();
    boiler.Poll();
 }
1 The 6 lines of code appear in the "CoffeeMaker::_Poll()" function.

If you are using a diagram as we are in this solution, you always change the diagram first when the requirements change. It provides the expressiveness needed to see the application’s requirements represented in a clear, concise and coherent way. There the logic can be ‘reasoned’ with. It is not documentation, it is the source code representation of the requirements, and executable, both important aspects of ALA.

The next step is to implement the three abstractions. These are straightforward using the same execution model as was used for the application, so are not shown here.

The resulting application passes all of Martin’s original acceptance tests plus a number of additional tests of behaviour gleaned from his original text.

5. Chapter five - Methodology

5.1. Agility

Apart from an iteration zero, ALA is inherently optimally agile. By agile, we mean it’s easy to change the functional requirements. ALA achieves this in its highest level separation of concerns. This primary separation is code that just describes requirements from code that just does implementation. The implementation code never has knowledge of any specific requirement, so it generally doesn’t change when requirements change. Only the code that describes requirements needs to change, and that code is optimally minimal over requirements, and so is optimally agile over changes to requirements.

5.1.1. Iteration zero

When a new project begins, the only new information we have is the requirements. Any design decisions that don’t depend on the requirement could already have been made beforehand. It is those decisions that form the ALA reference architecture. Therefore,, when we get the requirements, that is our immediate and total focus. We may not know all of them, but we will only need a sample to build a picture of the architecture.

Looking at the available new information as a whole first instead of taking it a bit at a time during the project’s sprints will make a huge difference to the eventual architecture quality.

The process in the first iteration takes requirements one by one, and represents them, in all their detail. Domain abstractions will be invented as you go, and they will have parameters or properties that will handle those details from requirements.

For the first green field application, you spend a maximum of one sprint. After that you do need to find out if your design works. So you may not get through all the known requirements. That does not matter.

To know whether knowledge from your design goes in the application layer of the domain abstractions layers, you consider what the scope of that knowledge is. Is the knowledge specific to this one requirement in the one application, or is it potentially reusable in the same or other applications? A softkey label is clearly specific. The concept of softkeys is clearly in the domain.

The output of the first sprint does not implement any of the invented abstractions, but it does include all details of the requirements that are looked at. In so doing, you design the first approximation of a DSL. The DSL may be refined later as more requirements are looked at.

Each abstraction will eventually be implemented as a class, but initially we just record the names of the abstractions, and a short explanation that provides the insight into what this abstraction does.

By the end of the first sprint the requirements will have become easier and easier to represent, as the set of abstractions will have taken shape. Sometimes you will generalize an abstraction further to enable it to be useful in for more things.

By keeping moving through the requirements at a much faster pace than in normal development (say one feature per hour instead of one per week), we can keep representing them in a coherent way, revising abstraction ideas we have already invented. Ideally, we will end up with a set of domain abstractions that can be wired together in different ways to represent a ‘domain space’ of requirements. That domain space will grow slightly as time goes on and it accommodates a growing family of products, but we don’t want it to grow beyond that. We don’t want to invent ways of implementing things we will never do. If you leave the invention of the set of abstractions to be done gradually during the longer time scale of the project as a whole, you will lose the opportunity to have a coherent set that will compose with each other in an infinite variety of ways.

The output of sprint zero is usually a diagram showing the wiring of instances of abstractions, together with annotated configuration information for those instances.

It doesn’t matter if some of the requirements are wrong. Chances are they are still useful for scoping out the domain space. What we are actually producing in this phase are the necessary abstractions for the Domain layer. If the requirements change later, it will be trivially simple to change them as only the Application layer wiring should change.

Once this process has started to become easy, which should happen within the first sprint, the burning question in our minds will become “Will all this actually work?” We have to trust that there will be a way to write those abstractions to make the whole system work.

5.1.2. 2nd Sprint

In the second sprint we start converging on normal Agile. You pick one feature to implement first. Agile would say it should be the most essential feature to the MVP (minimum viable product), but this can be tempered by the need to choose one that requires a fewer number of abstractions to be implemented. Next, you design the interfaces that will allow those abstractions to plug together according to the wiring you already have from the first sprint. What messages will these paradigm interfaces need to pass at runtime between the unknown clients to make them work?

It may take several sprints to produce the first working feature, depending on the number of abstractions it uses.

At first this sounds as if it might be just the waterfall method reincarnated. Do an overall design, document it or model it, and then write lots of code before everything suddenly starts working. But the design we created in iteration zero is very different from what a normal waterfall would produce, and is resilient to the sorts of problems waterfall creates. Instead of a ‘high level’ design of how the implementation will work which is lacking in detail, the design is a representation of requirements, in full detail. The design is not a model. It is executable.

There is one more important thing that the design phase in Iteration Zero does. While it deliberately doesn’t address any implementation, it does turn the remainder of the implementation into abstractions, and those abstractions are zero coupled. To convert from executable to actually executing, it only remains to implement these now completely separate parts. You can give these abstractions to any developer to write. Together the developers will also easily be able to figure out the paradigm interface methods needed to make them work, and the execution models to take care of the execution flow through them with adequate performance.

Often when a project is split into two phases, the first phase turns out to be waste. The devil is in the details so to speak. This happens because the implementation details in phase two are coupled back to and affect the design in phase one. As learnings take place during implementation, the design must change. In ALA the output from phase one is primarily abstractions, which are inherently stable and therefore hide details that can’t affect the overall design. If the abstractions are good, phase two will typically have little effect on the work done in phase one.

Once the first feature is working, several abstractions will have been implemented. The second feature will take less time because some of the abstractions are already done. In ALA velocity increases as time goes on and keeps increasing until new features only involve instantiating, configuring and wiring domain abstractions in new ways. This velocity acceleration is the complete opposite of what happens in monolithic code.

5.1.3. Later sprints

Imagine going into a sprint planning meeting with a Product Owner, a small team of developers, and a mature ALA domain that already has all the common domain abstractions done. As the Product Owner explains the requirements, one of the team members writes them down directly as they would be represented in terms of the domain abstractions. Another team member watches and remembers any lost details without slowing the product owner down. A third member implements the acceptance tests in similar fashion, and a fourth provides him with test data. It would be nice to have a tool that compiles the diagram into the equivalent wiring code. With such a tool, the team could have it executing by the end of the meeting. At the end of the planning meeting the development team say to the product owner "Is this what you had in mind?". The team can get immediate feedback from the Product Owner that the requirements have been interpreted correctly.

Of course, the planning meeting itself would only produce 'normal' functionality. Usually it is up to the development team, not the Product Owner, to uncover all the abnormal scenarios that can happen, and that is usually where most of the work in a software system goes. Having said that, in a mature domain, the validation of data already has decorator abstractions ready to go.

effort curve.png

The graph shows the effort per user story against months into a green-field project. The left axis is arbitrary - the shape of the curves is what is important. For a big ball of mud, experience tells us that the effort increases dramatically and can asymptote at around 2 years as our brains can no longer handle the complexity, and the project must be abandoned.

The COCOMO model, which is an average of industry experience, has a power relationship with program size, with an exponent of around 1.05 to 1.2. I have used the mid point, 1.1, for this graph. The model appears to imply that getting lower than 1.0 is a barrier, but there is no reason to believe this is the case. Reuse can make the power become less than 1. The range of 1.05 and 1.2 probably results from some reuse mitigating some ever increasing complexity.

ALA takes advantage of the fact that zero-coupled abstractions can keep complexity relatively constant and drastically increase reuse. A spectacular fall in effort per user story is thus possible.

5.2. Folder structure

[tree,file="folderstructure.png"]
root
|--Application1
|  `--application.cpp
|--Application2
|  `--application.cpp
|--DomainAbstractions
|  |--abstraction1.cpp
|  |--abstraction1.h
|  |--abstraction2.cpp
|  `--abstraction2.h
`--ProgrammingParadigms
   |--Paradigm1.h
   `--Paradigm2.h

This is a suggested folder structure for ALA. Because ALA does not use decomposition, you don’t end up with components that are contained by the applications, so there are no subfolders under the application. Instead, you end up with Domain Abstractions outside the application, so they go in their own folder in a flat structure.

Similarly, the Programming Paradigms code is not contained by an application, or even by the domain, so would not be contained by the domain’s projects folder.

5.3. Convention over configuration

When the application create an instance of an abstraction, most of the configuration of that abstraction should have defaults. In ALA, setters allow optional configuration, reducing the amount of information that would otherwise be required in the application to fully configure each abstraction. Any configuration that we wish to enforce goes into the constructor.

There is a counter argument that says that all configuration should be explicit so that nothing can be forgotten. ALA prefers optional configuration because we want the application to just express the requirements. Also optional configuration allows abstractions to default to their simplest form, making them easier to learn.

5.4. Knowledge prerequisites.

When other programmer are doing maintenance on your code, you should make sure they have the knowledge they need. They will need knowledge of ALA. They will need to know about the programming paradigms used. They will need to know about the domain abstractions, and the insight of what each one does. And then they should know that the application diagram is the source code. It is up to you that every develop that follows will know all this.

5.4.1. Intellisence

After they have modified the diagram, the maintaining developers will need to manually modify the corresponding code. Here they will see instances of abstractions being used all over the place, either 'new' keywords or function calls. If we have done our job with knowledge prerequisites, they will have been introduced to these abstractions. However, it doesn’t hurt to have brief reminders of what they are pop up when the mouse is hovered over them. So put in triple slash comments (or equivalent) describe the abstraction succinctly, with the intention of it being a reminder to someone who has already met the abstraction. Put a full explanation in the remarks and examples sections.

The class name after a new keyword is actually the constructor name, so you must duplicate the summary section there. Often in ALA, the class name is not referred to at all in the application.

5.5. Two roles

ALA requires two roles. Both can be done by the same person, but always he should be wearing only one hat at a time. There is the role of the architect, and the role of the developer.

The role of the architect is harder than that of the developer, that’s why we have the role. Expect it to be hard. Perhaps, surprisingly, the architect’s main job is to focus on the requirements, and the developers main job is to implement the abstractions (which know nothing of the requirements). In describing the requirements, the architect invents domain abstractions.

The architect also has a role in helping the developer to design the interface’s methods. In other words, how at runtime the system will be made to work.

This aspect of ALA can also be difficult at times. I have sometimes got stuck for a day or so trying to figure out how the interfaces should work, while still keeping them more abstract than the domain abstractions. The ALA constraints are that these interfaces should work between any two domain abstractions for which it may be meaningful if they are composed together in an application. However, the problem has always been solvable, and once solved, it always seems to have a certain elegance, as if you have created a myriad of possibilities at once. Implementation of the interfaces by the relevant domain abstractions becomes easy. Development then proceeds surprisingly quickly.

5.6. Example project - a real device

Unlike our previous example projects, this project is a real device and had previously been implemented without any knowledge of ALA. So this example serves to make comparisons between ALA and conventional software development. The original software was around 200 KLOC and took 3 people 4 years to write.

The actual device is used by farmers all over the world. It can weigh livestock and keeps a database about them for use in the field. It connects to many other devices and has a large number of features:

XR5000 image
Figure 18. Livestock weighing indicator

The architecture in the original software, was somewhat typically organised into modules and patterns by its developers. Also somewhat typically, it had ended up with a high cost of modifiability - a big ball of mud. After the first release, the first incremental requirement was a 'Treatments' feature, which involved several new tables, new columns in existing tables, new data pages, new settings pages and some new business logic. This feature took a further 3 months to complete (actually 6 calendar months), which seemed out of proportion for the size of the feature. Somehow the Product Owner and managers seemed to have a sort of intuition that if similar things had been done before, such as menus or database tables, those things were already done, and the only new work was in the specific details of the new requirements. Those requirements could be communicated in a relatively short time, say of the order of one hour or one day if you include follow up discussions of abnormal scenarios. So 6 months did not go down well. ALA, of course, works in exactly this intuitive way that managers hope for. All the things already done are sitting there in the domain abstractions, waiting to be reused, configured and composed into new requirements.

5.6.1. Iteration zero

During the development, there had a been a high number of changes required to the UI. It occurred to me at the time that the underlying elements of the UI were not changing. It was mainly the details of layout and navigation around the device’s many pages that were changing. The same could be said about the data and business logic. Only details were changing.

I took to representing the new designs using box and line drawings representing both the UI layouts and the navigation flows. I realized these diagrams were potentially executable, and wondered how far I could go representing the data and business logic in the same way. I decided to try to represent all of the functionality of the indicator in just one diagram.

It took two weeks to complete the diagram. I used Xmind because it laid itself out. I found that any drawing package that needed you to stop and do housekeeping such as rearranging the layout got in the way so much that you would lose your flow. Xmind allowed me to just enter in the nodes and it would automatically wire them in as either peers or chains and lay them out. The one disadvantage was that Xmind only does trees, so any cross tree relations had to be done manually, but this was also very quick in Xmind once you were used to it. I just let the cross wiring form arcs across parts of the tree.

Progress was extremely rapid once you had the abstractions and paradigms you needed. And many of them were obvious: softkeys, pages, grids, menus, actions, navigate-action, tables. etc. The programming paradigms would pop into play as needed. After the obvious UI-layout and navigation-flow ones came data-flow and data-flow of table types, events, and schema. The user of this device could set up custom fields, so the schema itself partially came from another table. At times I would get stuck not knowing how to proceed. The longest of these blocks was half a day. But every time the required abstractions or programming paradigms would firm up, and in the end anything seemed possible.

The diagram itself took shape on the right hand side of the Xmind tree. On the left side I had the invented domain abstractions and paradigm interfaces, with notes to explain them. The right side was mostly just a set of relatively independent features, but there was the odd coupling between them such as UI-navigation lines that were also present in the requirements.

The diagram contained around 2000 nodes (instances of the abstractions), which is about 1% of the size of the total original code. There were about 50 abstractions, and several paradigm interfaces.

Part of the diagram is shown below (laid out more nicely in Visio)

Application diagram for the All Animals View feature
Figure 19. Application diagram for the All Animals View feature

As I did the diagram, I deliberately left out anything to do with the aforementioned Treatments feature, so that I could see how easy it might have been to implement once the domain abstractions for the rest of the requirements had matured. So after the diagram was completed, I added the Treatments feature. This involved adding tables, columns to existing tables, a settings screen, a data screen, and some behaviours. No further abstractions needed to be invented. The incremental time for the diagram additions was of the order of one hour. Obviously testing would be needed on top of that, and the 'Table' abstraction would need additional work so it could migrate itself, a function it had not needed up until this point. Although somewhat theoretical, the evidence was that we could get at least an order of magnitude improvement in incremental maintenance effort.

At first the diagram seemed too good to be true. It had an elegance all of its own. It apparently captured all of the requirements, without any implementation at all, and yet seemed potentially executable. And if it worked, application modifications of all the kinds we had been doing were going to be almost trivial.

The burning question on my mind was, is it simply a matter now of writing a class for each of these abstractions and the whole job is done?

5.6.2. Translating the diagram to code

We hired a C++ student and proceeded with a 3-month experiment to answer this question.

It was a simple matter to translate the diagram into C++ code that instantiated the abstractions (classes), wired them together using dependency injection setters, configured the instances using some more setters, and used the fluent interface pattern to make all this straightforward and elegant. Part of the code for the diagram sample above is shown below to give you a feel for what it looked like.

m_animalListScreen
	->wiredTo((new Softkeys())
		->wiredTo((new Softkey())
			->setTitle("History")
			->wiredTo(new Navigate(m_animalHistoryScreen))
		)
		->wiredTo((skeyOptions = new Softkey())
			->setTitle("Options")
			->wiredTo(new Menu()
				->wiredTo(new Navigate("Session...", m_sessionSummaryScreen))
				->wiredTo(new Navigate("Settings...", m_settingScreen1))
			)
		)
	)
	->wiredTo((searchField = new TextDisplayField())
		->setLabel("Search")
		->setField(VIDField = new Field(COLUMN_VID))
	)
	->wiredTo(new Grid()
		->wiredTo(columnOrder = new ColumnOrder())
		->setRowMenus((new EventHandler())
			->setEvent(EVT_KEY_ENTER)
			->wiredTo(new Menu()
				->wiredTo(new Navigate("View information for this animal", m_animalSummaryScreen))
				->wiredTo((new Action("Delete Record", AnimalsTable::DeleteRow))->wiredTo(AnimalsTable))
			)
		)
	);

5.6.3. Writing the classes

We knew we wouldn’t have time to write all 50 classes, so we chose to implement the single feature shown below as a screen shot.

All Animals screen shot
Figure 20. All Animals view in the weighing indicator

The student’s job was to write 12 abstractions out of the 50. These 12 were the ones used by that feature. The initial brief was to make the new code work alongside the old code (as would be needed for an incremental legacy rewrite), but the old code was consuming too much time to integrate with, so this part was abandoned.

The learning curve for the student was done as daily code inspections, explaining to him where it violated the ALA constraints, and asking him to rework that code for the next day. It was his job to invent the methods he needed in the paradigm interfaces to make the system work, but at the same time keep them abstract by not writing methods or protocols for particular class pairs to communicate. It took about one month for him to fully 'get' ALA and no longer need the inspections.

The student completed the 12 classes and got the feature working in the device. The feature included showing data from one database table in a grid, sorting, searching, softkeys, and a menu.

Interestingly, as the student completed certain abstractions that allowed parts of other features to be done, he would quickly go and write the wiring code and have the other features working as well. For example, after the softkeys, actions, navigate, and page abstractions were done, he went through and added all the softkey navigations in the entire product as this only took minutes to do.

We wanted more funding to retain the student until we had enough to do the treatments feature, and indeed all 50 abstractions with the hope of making this implementation the production code and improving our ongoing maintenance effort. But that was not to be, despite the promising result.

We have about a quarter of a data point. Some of the abstractions done were among the most difficult, for example the Table abstraction, which had to use SQL and a real database to actually work. So it is not unreasonable to use extrapolation to estimate that the total time to do all 50 abstractions would be about one person-year. That compares with the original 12 person-years.

It seems that classes that are abstractions are faster to write. This seems intuitive because you don’t have any coupling to worry about. More importantly, the two phase design-then-code methodology of ALA allows the developer not to have to deal with large scale structure at the same time as writing code. This frees the developer to go ahead and write the code for the local problem.

I believe it is beneficial for each developer to be trained to be both an architect and a developer, but just don’t ask them to do both at the same time.

This practical result combined with the theory outlined earlier in this article suggests there ought to be a large improvement in incremental maintenance effort over a big-ball-of-mud architecture.

6. Chapter six - The philosophy behind ALA

6.1. The human brain

In this perspective of ALA, we look at the problem of complexity in software in the context of how the human brain works.

Software design involves our intelligence or brain power. Understandability, readability, complexity are all things very closely related to the brain. Yet in the field of software engineering we pay little attention to how the brain understands our complicated world in order to understand how we should do our software.

Our brains do it primarily through one mechanism which we have come to call 'abstraction'. We learn abstractions from the commonality of multiple examples, and we then use abstractions without those examples cluttering up the common notion that was learned.

Paintings from the Chauvet cave.jpg

Our ancestors could use a word like 'bring your spear' and it had a simple meaning to them only because all the detail and knowledge that went into building a spear was replaced with the abstraction 'spear'. Without the abstraction, the sentence would have had to be more like "bring the object that we made by joining the object we made by applying blows to the hard material we found at the place…​, with the long material we cut in the place with the tall…​, by tying it with the long grass material using the gooey stuff we found at the…​". Even this sentence was only made possible by other abstractions: joining, material, blows, hard, long (twice), cut, tall, tying, gooey, and found. If we expanded all of them until we were only using a few basic concepts like 'object' and 'place', we would have a sentence so long that we could never communicate at all. That’s what abstractions do, and how our brains make use of them. The word spear, in turn can be used to create a new abstraction, a hunting plan, while all of that other detail remains hidden from that new context.

The problem with software engineering is we are not making use of this way that the brain works. Simply put, we are not creating good abstractions. This lets the complexity inside one 'module' spill out into other modules. Abstractions, not modules, are the only mechanism that allows us to hide details at design-time.

neuron.svg

As software engineers we do learn and use many abstractions. For example if we want to protect simultaneous access to the resource, our brain should conjure up 'mutex'.

If the brain already 'knows about' an abstraction, the abstraction is like any other single line of code, such as a mutex. We can make use of a mutex without having to deal with the details and complexities of how it works. We don’t have to think about the fact that we may have to wait for the resource. Nor that another thread may start running if we have to wait. Nor that if a higher priority thread preempts us while we have the resource, we may have it for a long time. Nor that if a still higher priority thread needs it during that long time, we will be given temporary priority to finish our use of it. We can just simply use the abstraction for protecting a resource.

Abstractions like mutex, regex, SQL are already invented by the 'culture' of software engineering, much like memes in the real world have been passed down to us. Where we fall down is when we get into a particular domain where the abstractions have not yet been invented, and we need to invent them. It is not easy to invent new abstractions, but invent them we must, at a rate far higher than is normal for cultural evolution.

Good domain abstractions, introduced and learned by new developers in the domain, then appear to them as normal program elements - things they can use with extraordinary convenience like any other line of code.

6.2. Abstraction

Of the overwhelming list of engineering topics that we listed in Chapter One, this topic that is the most fundamental to ALA, and the one most needed for explaining it. It’s also probably the vaguest and most misunderstood topic in software engineering, so we will spend some time understanding it.

Abstraction will be the king. The short reason why we start with abstraction is that our quality attributes, complexity and understandability are very much to do with how our brains work, and for 100,000 years at least, our brains have worked with abstractions to understand our world. Abstractions are the only mechanism our brains use for dealing with otherwise complex things.

As in a chess game, winning is only about protecting the king. But this Abstraction king is benevolent. If he is destroyed, you do not lose the game immediately. It will take time, but you will lose.

There are other contenders to be king in the engineering topics list. For example, it is said that the best thing about TDD is not the testing but the emergence of better abstractions. TDD is like a lord that serves the king. It usually serves the king, causing you to make better abstractions. But sometimes it just serves its own purpose and makes the abstraction worse. It just produces code that works where it passes and no more.

Another contender is microservices. It is popular because it improves your abstractions by making them harder to destroy with cross coupling. But it too is just a lord. Because it provides physical boundaries that in normal software would be crossed, it serves the king. But by serving the abstraction king directly we can have logical boundaries, and all their benefits, even in 'monolithic' code.

Another contender to be king is 'no side effects' used by the functional mathematical purity guys. There are those who talk as if disobeying this king is absolute treason. But again, this lord is only effective because he usually serves the abstraction king. But, again, there are times when he doesn’t, and 'no side effects' is not enough to make a good abstraction.

ALA always follows the one true king.

6.2.1. Classes, Modules, functions and encapsulation.

Classes, Modules, functions and encapsulation are artefacts of the language and do their thing at compile-time. They are not necessarily abstractions. They have been around for about 60 years, not enough time for our brains to see them in the same way as the compiler does. Although abstractions are implemented using these artefacts, ALA needs them to also be abstractions. In ALA "abstraction" is the term we use for the artefacts of our design instead of classes, modules, functions, or components, all of which are extremely fragile as abstractions.

6.2.2. Wikipedia on abstraction

"Thinking in abstractions is considered by anthropologists, archaeologists, and sociologists to be one of the key traits in modern human behaviour, which is believed to have developed between 50 000 and 100 000 years ago. Its development is likely to have been closely connected with the development of human language, which (whether spoken or written) appears to both involve and facilitate abstract thinking."

In the real world, new abstractions come along infrequently, and are conceived of by few. People quickly begin using them to understand new insights or compose new things. They become so natural to us that we forget that they are abstractions. In no other field do we need to create them as fast as in software engineering. It is the most important skill a developer needs to have.

6.2.3. Defining abstraction

The term abstraction is arguably one of software engineering’s vaguest and most overloaded terms. For the purpose of this article we will need a definition. I find the easiest way to define it is to provide a set of 'statements about', 'qualities of', and 'what it is nots':

  • What our brains evolved to use to understand things

  • Etymology: 'to draw out commonality'

  • The concept or notion represented by the commonality of many instances

  • Has inherent stability - as stable as the concept itself

  • Increases with ubiquity and reuse

  • Decreases as you get closer to your specific application

  • The only mechanism that separates and hides knowledge at design-time

6.2.4. The three stages of creativity

Creativity with abstractions
Figure 21. The creativity cycle

A good abstraction separates the knowledge of different worlds. A clock is a good abstraction. On one side is the world of cog wheels. On the other side is someone trying to be on time in their busy daily schedule. Neither knows anything about the details of the other. SQL is another good abstraction. On one side is the world of fast indexing algorithms. On the other is finding all the orders for a particular customer. Let us consider a domain abstraction - the calculation of loan repayments. On one side is the world of mathematics with the derivation and implementation of a formula. On the other, the code is about a person wanting to know if they can afford to buy a house. If your abstractions don’t separate knowledge of different worlds like this, then you are probably just factoring common code. Find the abstraction in that common code. Make it hide something complicated that’s really easy to use and really useful, like a clock.

The creativity cycle starts with abstractions, such as cogs and hands, instantiates them, configures them for a particular use, then composes them into a new abstraction. In ALA we usually go around the creativity cycle three times, creating three layers on top of our base programming language.

6.2.5. An illustration of abstraction at work

Imagine you are reading the following function, xyz123, at design-time and trying to understand it:

real xyz123(real)
{
    ...
    b = fubar(a)
    ...
}
real fubar(real)
{
    // complicated code
}

You don’t know what fubar is (fubar stands for messed up beyond all recognition), so you follow the indirection, an inconvenience at the least because you are really just interested in xyz123. You begin reading the code at fubar. It only has about 20 lines but it is complicated. A comment mentions that it uses a CORDIC algorithm and gives a reference. But before following that indirection as well, you note that fubar has the following properties:

  • a module

  • has a simple interface

  • encapsulated

  • uses nothing but what is in the interface

  • no side effects

  • hides information

  • is loosely coupled

  • separates two concerns

  • is small

  • follows the coding guidelines

  • has comments

It could even have a good name describing what it returns in terms of its input, something like Returns X such that X is …​., but that still wouldn’t solve the issue.

Why, if it has all these good software properties, is it destroying our ability to understand function xyz123? The missing ingredient is of course abstraction, the thing the brain needs for meaning. So let’s go ahead and add the abstraction property. To do that, change fubar to just 4 letters (it doesn’t even need to be a good name): SQRT. While we are at it, let’s add the abstraction property to xyz123 as well, by naming it StandardDeviation.

real StandardDeviation(real)
{
    ...
    b = SQRT(a)
    ...
}
real SQRT(real)
{
    // complicated code
}

Suddenly understandability in the first function is unlocked. The complicated code inside SQRT no longer matters. It is completely isolated by the abstraction. If your brain already knows the SQRT abstraction (I had to choose one that I knew you already knew), there is no need to follow the indirection. The reader of the code continues reading with the next line of code after the SQRT invocation as if it is just like any other line of code in their language. That’s what abstraction, at least as used in ALA, is.

All those other properties made no difference while the abstraction property was missing. But any one of them could destroy the abstraction. Our current programming languages provide nothing to help us with abstraction. At least for the meantime, it remains the one cerebral activity software engineers must do for themselves.

6.2.6. Abstractions own their outputs

In traditional architecture, functions, modules or classes obviously have inputs and outputs.

In ALA, outputs to abstractions in a lower layer should be done in the same way as they have always been done. Input from a lower layer will require a dependency inversion such as the observer pattern.

Inputs and outputs to peer abstractions need to be done differently.

Inputs that are static functions of a module are fine, but should be clearly grouped with any other inputs that would be expected to come from the same peer.

Inputs and outputs of a class should not be in the class’s main interface. The class’s interface should only be used by the abstraction in a higher layer to instantiate and configure the abstraction. This is the interface segregation principle. A class should have a new interface for each I/O port that can be wired to a peer. These interfaces are abstractions from a lower layer and should represent a programming paradigm.

Programming languages encourage outputs to peers to be implemented as direct function calls, or method calls on an interface owned by another abstraction. If you are using an asynchronous event driven design, it is common for an output to be written like this:

Send(receiversEventID, receiverID, priority);

In ALA, sending an event would be written similar to this:

Send(senderEventID, senderID, senderPortID)

The abstraction shouldn’t know the (shared) event, the destination or the priority. An abstraction in a higher layer has the application specific knowledge about the wiring and will know the destinations, the priority, and the receiver’s event.

In general, classes, modules, components, functions should own their outputs. Say what this output is in term of its own abstraction. 'This has happened'. 'Here is my result'.

Outputs that will use synchronous function calls or methods calls must similarly be wirable abstract outputs that don’t know anything about where they go. Function calls could go via a callback or a signal & slot library. Whole interfaces of methods go via an abstract interface from a lower layer, injected in by a higher layer abstraction.

Note that inputs and outputs are not necessarily different ports. We may want to wire both input and outputs between two abstractions with a single wiring operation. This is the Interface Segregation Principle. The general case is that a single wiring operation wires a pair of interfaces that are logically one interface. One contains methods going in one direction and the other contains methods going in the other.

6.3. Complexity

6.3.1. Philosophy of complexity

6.3.2. Dijkstra on complexity

"It has been suggested that there is some kind of law of nature telling us that the amount of intellectual effort needed grows with the square of program length. But, thank goodness, no one has been able to prove this law. And this is because it need not be true. We all know that the only mental tool by means of which a very finite piece of reasoning can cover a myriad cases is called “abstraction”; as a result the effective exploitation of his powers of abstraction must be regarded as one of the most vital activities of a competent programmer. In this connection it might be worth-while to point out that the purpose of abstracting is not to be vague, but to create a new semantic level in which one can be absolutely precise. Of course I have tried to find a fundamental cause that would prevent our abstraction mechanisms from being sufficiently effective. But no matter how hard I tried, I did not find such a cause. As a result I tend to the assumption —up till now not disproved by experience— that by suitable application of our powers of abstraction, the intellectual effort needed to conceive or to understand a program need not grow more than proportional to program length."

The "conceive" part I agree with, if by that we mean the development. However, the "intellectual effort to understand" part needs further insight. We shouldn’t have to read an entire program to understand a part of it. We ought to be able to understand any one part of it in isolation. The effort to read any one part should be approximately constant. In Chapter One of this article there was a quality graph of complexity here.

complexity curve.png

These graphs are qualitative in nature, based on experience. But now that we have a better understanding of ALA structure, we can explain how it manages to keep complexity from increasing.

In ALA, the design-time view of the system is nothing more than a static view of instances of abstractions composed together. In a typical application, there will be of the order of fifty different domain abstractions - not a difficult number to familiarize yourself with in a new domain.

Abstractions have no relationship with one another. Each is a standalone entity like a standalone program. If every abstraction contains say 500 lines of code, and the system itself contains 500 lines (instances of abstractions wired together) then the most complex the software gets is that of 500 lines of code.

Even if one abstraction is overly complex internally, say it conceals a piece of legacy code using a facade pattern, that doesn’t affect the complexity of any other part of the system.

ALA is based on the realization that abstraction is fundamentally the only mechanism available to us to achieve this constant complexity.

When doing this for the first time in a domain, it’s not easy to invent the abstractions. but the alternative is always runaway complexity.

The goal of software architecture should be to keep complexity constant.

6.4. No Loose Coupling

Here we meet the first meme from our list of software engineering topics that we must throw out. To many, this will seem a surprising one. Yes, I am saying 'loose coupling' is undesirable.

6.4.1. A common argument

An argument is sometimes stated along these lines: "There must be at least some coupling, otherwise the system wouldn’t do anything." Hence we have the common meme about "loose coupling and high cohesion". In this section we show how this argument is false and resolve the apparent dilemma. We will eliminate all forms of design-time coupling except one. That one remaining one is anything but loose and very desirable.

6.4.2. Classifying coupling

Think of some instances of dependencies you know of in a system and try to classify them into these three types by asking when the system would fail without it.

For example, let’s say that data flows from an ADC (analog to digital converter) to a display as part of a digital thermometer. At run-time, both must exist. At compile-time both must have the same method signature:

diagram 01.png

Or the display may tell the ADC when to do the conversion. At run-time there is temporal coupling.

diagram 02.png

In this one there is an association from a Customer class to an Account class to facilitate communication between them. At run-time there is coupling. At compile-time there is coupling too - the type of the Account class must be exactly the same as expected by the Customer class:

diagram 04.png

In all the above diagrams, relationships shown in red indicate they are disallowed by the ALA constraints. Green is for desirable relationships, of which there is only one. When we disallow all these types of coupling, the modules, components, functions and classes can now be abstractions.

6.4.3. Run-time, Compile-time and Design-time

A few times already in the article, I have sneaked in a magic qualifier, 'design-time'. You know how we sometimes talk about run-time and compile-time with reference to binding. In ALA we recognise that understandability, complexity, etc, are all happening at design-time. By design-time I mean any time you are reading code, writing code, or changing code.

At run-time, the CPU processes data. At compile-time, the compiler processes code. At design-time the brain is processing abstractions.

In conventional code, it is common for all forms of coupling, run-time, compile-time, and design-time, to appear as coupling between modules or classes.

You can work out what type of dependency you have by when it first breaks. A run-time dependency doesn’t break until the program runs. The program can still be compiled and it can still be understood.

A compile-time dependency first breaks at compile-time. At design-time the code can still be understandable.

A design-time dependency prevents code from even being understood. The code loses its meaning.

6.4.4. Zero coupling

When we say no loose coupling, it means there is zero coupling between the details and knowledge contained in any two abstractions. Abstractions are free floating little independent programs.

Since our conventional programs are typically full of coupling of all sorts, this constraint on the architecture will obviously change how we write programs significantly. But surprisingly, things quickly get easier, not harder with these constraints.

6.4.5. Knowledge dependencies

The one form of coupling allowed in relation to abstractions is from the hidden code of one abstraction to another abstraction. It is always one directional. Because it is always directional, we call it a dependency rather than coupling. From now on it will be referred to as a 'knowledge dependency'. Elsewhere it is referred to as semantic coupling. Since it’s the only relationship we have between abstractions, it’s obviously important and we will be giving it a lot of attention in following sections. It will drive the layering structure, which in turn gives rise to the name 'Abstraction Layered Architecture'.

A knowledge dependency is the code implementation of one abstraction requiring knowledge of another abstraction to be understood. Without the knowledge, the code using the abstraction would lose meaning and no longer make any sense to a human reader. It could not even be written in the first place.

6.4.6. Knowledge dependencies always go down

In ALA, knowledge dependencies must be from inside an abstraction that is less abstract to one that is more abstract. We will always put abstractions that are more abstract lower down in the layers. In everyday design, knowledge dependencies are not normally drawn. You simply use the abstraction by its name. But in this article, just so we can explain the meta-architecture, we will sometimes draw knowledge dependencies like this (always downward).

diagram 05.png

This represents that the implementation of abstraction C knows about abstraction A. A is more abstract than C. C and A cannot therefore be peers, as was the case with the components above. Peer abstractions cannot have any coupling with one another.

6.4.7. Whole-Part pattern

If you are familiar with the Whole-Part pattern, ALA uses it extensively. But there is a constraint. The Whole-Part pattern is only used with knowledge dependencies (since that is the only relationship you are allowed). It may of course be used in other forms inside an abstraction, provided it is completely contained in a single abstraction.

A real world example of the Whole-Part Pattern with knowledge dependencies is Molecules and Atoms. A water molecule, for example, is the whole.

diagram 06.png

Oxygen and hydrogen are the parts. Note that oxygen and hydrogen are abstractions, and they are more abstract than water because they are more ubiquitous, more reusable and more stable (as a concept) than any specific molecule. We could make a different molecule but still use exactly the same oxygen and hydrogen as parts to compose the new molecule.

When we use the word 'ubiquitous', it refers to the number of times the abstraction is used in a Whole-Part pattern to make other abstractions. It doesn’t refer to the number of abstractions that are instantiated. So just because there is a lot of water, that doesn’t make the abstraction ubiquitous. In comparing the abstraction levels of Oxygen and Hydrogen with water, Oxygen and Hydrogen are more ubiquitous because they are used to make more abstractions than water is.

The molecules and atoms analogy with ALA is very close, and we will return to it when we come to explain in more detail how run-time and compiler-time dependencies are moved inside a single abstraction.

For now we just need to remember that we are using the whole-part pattern with knowledge dependencies only. At design-time, the whole is explained and reasoned about in terms of the parts, just as the water molecule is in terms of the oxygen and hydrogen.

6.4.8. Run-time/design-time congruence

A software program can be temporally confusing. Everything that happens at design-time is in preparation for what will happen at run-time. Our low-level imperative languages tend to keep the two congruent. The statements in the program at design-time follow in the same order as they will execute at run-time. The only difference between the two is a time shift and the speeding up of the clock.

When we want the knowledge of run-time dependencies to be moved inside another abstraction, this congruence between design-time and run-time must be broken. Unfortunately, developers start out by learning a low-level imperative language, so it becomes unnatural to them to architect their programs without this congruence. Indeed, breaking this congruence needs a pattern to be learned, and then carefully protected from the temptations of our imperative languages. I call it the Ẃiring pattern'.

Before going into the pattern, we need to round out the most important aspects of ALA.

6.4.9. Wiring pattern - Part one

We now introduce the pattern that both solves the congruence problem just discussed in the previous section, and provides the alternative to all those disallowed coupling types discussed earlier. This pattern is usually an important part of ALA.

Note: The wiring pattern is not necessarily a part of an ALA architecture. For example, if your whole problem is just an algorithm, and therefore suits a functional programming style, then you can still compose abstractions with function abstractions, provided all function calls are knowledge dependencies, and not say, just passing data or events.

If you are using monads, especially I/O monads, or RX (reactive extensions), especially with hot observables, you are already using the wiring pattern. The pipes and filter pattern is also an example of the wiring pattern. Labview or Node-Red can use the wiring pattern. There are many other examples of the wiring pattern. Most support a data-flow programming paradigm. Here we generalize the pattern to support any programming paradigm.

The wiring pattern may be the same as the "Component Pattern" in some literature if used with what is referred to as 'configurable modularity' or 'abstracted interactions'.

The wiring pattern allows lines on your application diagram to mean any programming paradigm you want that express your requirements. It also allows you to implement multiple programming paradigms together in the same diagram.

If you are using dependency injection with explicit code for the wiring (not auto-wiring), then you are half way there.

The wiring pattern separates design-time/run-time congruence. It works by having a 'wiring-time' that is separated from run-time. 'Wiring-time' can happen any time before run-time. It can happen immediately before it, as for instance in LINQ statements or RX with a cold observable. It becomes powerful when we make wiring-time congruent with design-time. Usually the wiring code will actually run at initialization time, when the program first starts running. That initialization code becomes the architectural design.

Let’s suppose you have designed your system with two modules, A and B. There will be one of each in your system.

diagram 07.png

At run-time we know that A will talk to B. So we design A to have an association with B. The association may first appear on a UML model, or it may go straight into the code something like this:

static component A
{
   B_method();
}
static component B
{
   public B_method() { }
}

A and B may be implemented as non-static, with only one instance of each. The association is still there.

component A
{
   private var b = new B();
   b.method();
}
component B
{
   public method() { }
}

A may create B itself, which is a composition relationship, as above. Or A may have a local variable of type B passed in by some kind of dependency injection, which is still an association relationship.

component A
{
   B b;
   public setter(B _b) {b = _b}
   b.method();
}

Note that although dependency injection was used, it only eliminated part of the dependency, that of which particular subtype of B it is going to talk to, but A still knows the general type B, which is not allowed in ALA. (Part of the problem here is that A and B were probably arrived at by decomposition, and so they have subtle knowledge of each other, for example of how they collaborate.)

If A and B are collaborating, they are not abstractions. Their knowledge of each other at design-time (to enable their relationship at run-time) binds them to each other so that neither can be reused in any other way. And if they can’t be reused, they can’t be abstract.

Let’s revisit the water molecule analogy we discussed earlier for the Whole-Part pattern, and develop it further to be clearer how these dependencies affect abstractions. Let’s say we have decomposed water into two components, Oxygen and Hydrogen. Oxygen will talk to Hydrogen to get an electron, so we write:

component Oxygen
{
   var h1 = new Hydrogen();
   var h2 = new Hydrogen();
   h1.getElectron();
   h2.getElectron();
}

The diagram for that looks like this:

diagram 08.png

In the real world, oxygen is a very useful abstraction for making other molecules. In writing code this way to make water, we have tied it to hydrogen. Oxygen can’t be used anywhere else, at least not without bringing with it two hydrogens, rendering it useless. By implementing the Oxygen-Hydrogen relationship needed to make water in oxygen, we have destroyed the oxygen abstraction. We never even made the water abstraction. To understand water, we would have to read the code inside oxygen, where the parts about water have become entangled with the inner workings of oxygen, protons and neutrons and all that stuff. Oxygen is also used to make caffeine. We could never make coffee!

Caffeine molecule
Figure 22. caffeine - oxygen atoms are red

Abstractions are fragile and get destroyed easily, so we have to take care to protect them. What we needed to do was to put the knowledge about the relationship between oxygen and hydrogen to make water in a new abstraction called Water.

diagram 09.png

In general, to break coupling between peer modules A and B, we move the knowledge of the coupling to a higher level abstraction (less abstract level) where it belongs. Let’s call it C. C is a more specific abstraction. The knowledge is encapsulated there - it never appears as a dependency of any kind. And it is cohesive with other knowledge that may be contained inside abstraction C.

diagram 10.png

becomes

diagram 11.png

The diagram above is only to show the ALA knowledge dependency relationships between the three abstractions. It doesn’t yet show explicitly that an instance of Abstraction A will be wired to an instance of Abstraction B. In practice we never actually draw knowledge dependencies. We are just doing so here to show how ALA works. We would draw it in this way instead:

diagram 12.png
diagram 13.png

Now we have the explicit wiring. It looks a lot like the original diagram where we had no C. But where the knowledge is coded is very different. Because it is C and not A that has the knowledge of the relationship between A and B, Abstractions A and B do not change. They continue to know nothing of the connection. They remain abstractions. They remain re-usable.

It may seem at first that adding the extra entity C is a cost, but in fact C is an asset. It shows the structure of the system. It shows it explicitly. It shows it in one small understandable place. And it is executable - it is not a model.

The original abstractions were left below C to show that they still exist as free abstractions to be used elsewhere. They are not contained by C in any way as modules from a decomposition process would be. The A and B inside C are only instances. We wouldn’t normally bother to draw the abstractions below. So we just draw this:

diagram 14.png

C must achieve the connection between A and B either at compile-time or run-time. With current languages, the easiest time to do this is at initialization time, when the program first starts running. This is similar to dependency injection, except that we are not going to inject the instance of B into A.

This is what the code inside C might look like:

Abstraction C
{
   var a = new A();
   var b = new B();
   a.wireTo(b);
}

Typically we will write the code using the fluent pattern, with the wireTo method always returning the object on which it is called, or the wireIn method always returning the object wired to. The constructor already returns the created object by default.

Abstraction C
{
   new A().wireTo(new B());
}

If A and B are static modules, this produces something like:

Abstraction C
{
   A_setcallback(B_method);
}

6.4.10. Wiring pattern - part two

We are half-way through explaining the wiring pattern. Now we turn our attention to how A and B can communicate without knowing anything about each other.

This part of the pattern is also called "Abstract Interactions"

Of course, one way is that C acts as the intermediary. This way is less preferred because it adds to C’s responsibilities. But it is sometimes necessary if there are some abstractions brought in from outside. Such abstractions will 'own' their own interfaces or may come with a contract which C will have to know about. C will usually have to wire in an adapter, or handle the communications between the two abstractions itself.

A better way, because it leads to an architectural property of composability, is that A and B know about a 4th abstraction that is more abstract than either of them. This is legal because it is a design-time knowledge dependency. Let’s call it I.

diagram 15.png

I is an interface of some kind. It may or may not be an actual artefact. What it must be is knowledge that is more abstract than A and B and therefore knows nothing of A and B. It is more ubiquitous and more reusable than A and B are. In other words we can’t just design I to meet the particular communication needs of A and B. That would cause A and B to have some form of coupling or collaboration with each other, and again destroy them as abstractions.

I is so abstract, ubiquitous and reusable, that it corresponds to the concept of a programming paradigm. We will cover programming paradigm abstractions in following sections because they are a critically important part of ALA. We will see that ALA is polyglot with respect to programming paradigms.

Schematic
Figure 23. In an electronic schematic, the components are abstractions that are composed using two paradigm interfaces - live analog signals and live digital signals

Returning to a software example, let’s choose a single simple programming paradigm: activity flow. This programming paradigm is the same as the UML Activity diagram. When we wire A to B and they use this paradigm, it means that B starts after A finishes. If A and B accept and provide this interface respectively, then wiring them together by drawing an arrow will have that meaning, and cause that to happen at run-time.

diagram 16.png

It is easy to create an interface for the activity-flow programming paradigm. It has a single method, let’s call it 'start'. Many abstractions at the level of A and B can either provide or accept this paradigm interface. Then instances of them can be wired up in any order and they will follow in sequence just like an Activity diagram.

Note that the Activity Diagram is not necessarily imperative in that any Activity can take an amount of time to complete that is not congruent with the actual CPU execution of code. In other words activities can be asynchronous with the underlying code execution, and for example, delay themselves during their execution, or wait for something else to finish, etc.

The code in Abstraction A could look something like this. Don’t take too much notice of the exact method used to accomplish the wiring. There are many ways to do this using only knowledge dependencies. The important thing is that A continues to know nothing about its peers, continues to be an abstraction, and yet can be wired with its peers to take part in any specific activity flow sequence:

 Abstraction A : IActivity
 {
    private IActivity next = null;

    public IActivity wireTo(IActivity _next)
    {
        next = _next;
        return _next;
    }

    IActivty.start()
    {
        // start work
    }

    // code that runs when work is finished.
    // may be called from the end of start, or any time later
    private finishedWork()
    {
        if (next!=null) next.start();
    }
 }

Abstraction A both provides and accepts the interface. This allows it to be wired before or after any of its peer abstractions. In ALA we use the word 'accepts' rather than 'requires' because there is often an end to a chain of abstraction instances wired together. If no next interface is wired in, the activity flow ends.

Abstraction B would be written in the same way, as it also knows about the Activity flow interface:

 Abstraction B : IActivity
 {
    private IActivity next = null;

    public IActivity wireTo(IActivity _next)
    {
        next = _next;
        return _next;
    }

    IActivty.start()
    {
        // start work
    }

    // code that runs when work is finished.
    // may be called from the end of start, or asychronously later
    private finishedWork()
    {
        if (next!=null) next.start();
    }
 }
As an aside, in C# projects, we wrote wireTo as an extension method for all objects. It used reflection to look at the private interface variables in the source class and the interfaces provided by the destination class. It would then match up the interface types and do the wiring automatically. It could even use port names to explicitly wire ports of the same types.

Now let’s revisit the molecule analogy. By now we would know to put the knowledge that Oxygen is bonded to two Hydrogens inside the water abstraction where it belongs.

diagram 17.png

In terms of knowledge dependencies it means this:

diagram 18.png

The programming paradigm here is a polar bond. It is more abstract (more ubiquitous and reusable) than any particular atom. We could have a second programming paradigm, a covalent bond, as well. Again, the important thing here is not what the code does - that is arbitrary (and not actually correct chemistry) but how the atoms can be made to interact while retaining their abstract properties with only design-time knowledge dependencies:

Abstraction PolarBond
{
   GiveElectron();
}
 Abstraction Oxygen
 {
    private PolarBond hole1 = null;
    private PolarBond hole2 = null;

    public Oxygen wireIn(PolarBond _pb)
    {
        if (hole1==null) hole1 = _pb; else
        if (hole2==null) hole2 = _pb;
        return this;
    }

    public Initialize()
    {
        if (hole1!=null) { hole1.getElectron(); BecomeNegativelyCharged(); }
        if (hole2!=null) { hole2.getElectron(); BecomeNegativelyCharged(); }
    }
 }
 Abstraction Hydrogen : PolarBond
 {
    PolarBond.getElectron()
    {
        BecomePositivelyCharged();
    }
 }
 Abstraction Water
 {
    new Oxygen()
        .wireTo(new Hydrogen())
        .wireTo(new Hydrogen())
        .Initialize();
 }

Let’s do one more example, this time with a Data-flow programming paradigm. I have found that data-flow is the most useful programming paradigm in practice. It is useful in a a large range of problems.

Let’s construct a thermometer. Assume we already have in our domain several useful abstractions: an ADC (Analog Digital Converter) that knows how to read data from the outside world, a Thermistor abstraction that knows how to linearise a thermistor, a Scale abstraction that knows how to offset and scale data, a filter abstraction that knows how to smooth data, and a display abstraction that knows how to display data.

All these domain abstractions will use the Data-flow programming paradigm. Note that none of them know anything about a Thermometer, nor the meaning of the data they process.

So we can go ahead and create a Thermometer application just by doing this:

diagram 19.png

Note that we configure all the abstraction instances for use in the Thermometer by adding configuration information into rows on the instances.

When we manually compile the diagram (assuming we don’t have automated code generation), it might look something like this (again using fluent coding style):

Abstraction Thermometer
{
   new ADC(Port2, Pin3)
       .setFrequency(1000)
       .wireTo(new Thermister().setType('K').setInputRange(20,1023)
           .wireTo(new Scale(32,0.013)
               .wireTo(newDisplay().setDigits(4).setDecimals(1))
           )
       );
}
The configuration setters and the WireTo extension method return the object on which the call is made to support the fluent coding style.

The diagram is the requirements, the solution and the architecture of the application, and is executable. The diagram has all the cohesive knowledge that is a thermometer, and no other knowledge.

The diagram can be read stand-alone, because all the dependencies in it are knowledge dependencies on abstractions we would already know in the domain.

Let’s say when the Thermometer runs, there is a performance issue in that the ADC is producing data at 1kHz, and we don’t need the display to be showing Temperatures at that rate. Also the temperature readings are noisy (jumping around). Let’s make a modification to the Thermometer by adding a filter to reduce the rate and the noise:

diagram 20.png

If the domain abstractions are not already implemented, we have got the architecture to the point where we can ask any developer to implement them, provided we first give them knowledge of ALA and of the programming paradigm(s) being used.

But let’s look how the data-flow paradigm might work.

If you are familiar with RX (Reactive extensions) with a hot observable source (which is an example of the wiring pattern), this is similar in concept although RX tries to have duality with for-loops iterating through the data. The data-flow paradigm we set up here will just be a stream of data. The IDataFlow interface corresponds to IObserver, and the wireTo method corresponds to the Subscribe method.
The ideal would be a language where we don’t have to decide if the data-flow will be push or pull, synchronous or asynchronous, buffered or unbuffered or other characteristics of communications. The abstractions would not need to know these things - they would just have logical I/O ports, and the type of communications could be binded in at compile-time as part of the performance configuration of the system.
Later we will introduce an asynchronous (event driven) execution model. It is preferable to do the data-flow paradigm interface using that because it allows better performance of other parts of the system without resorting to threads.

For simplicity, we will just implement a synchronous push system. Again, don’t worry about the filter itself. The code is just there to see how the LowPassFilter fits in with the Data-flow programming paradigm, and how simple doing that can be.

Interface IDataFlow<T>
{
   push(T data);
}
 /// LowPassFilter is a Data-Flow paradigm decorator to be used in an ALA archtecture.
 /// 1. Decimates the incoming data rate down by the setCutoff configuration
 /// 2. Smooths the data with a single pole filter with cutoff frequency equall to the input frequency divided by the cutoff. T must be a numeric type.
 /// Normal checks and exceptions removed to simplify
 Class LowPassFilter<T> : IDataFlow<T>
 {
    private Dataflow next;

    // This is normally done by a general extension method
    public IDataflow wireTo(IDataflow _next)
    {
        next = _next;
        return _next;
    }

    integer cutoff;

    setCutoff(integer _cutoff)
    {
        cutoff = _cutoff;
    }

    int count = 0;
    T filterState = NAN;

    IDataFlow.push(T newData)
    {
        if (filterState==NAN) filterState = newData * cutoff;
        filterState = filterState - filterState/cutoff + newData;
        count++;
        if (count==cutoff)
        {
            count = 0;
            if (next!=null) next.push(filterState/cutoff);
        }
    }
 }

You will notice that both the Domain abstraction, Filter, and the Programming Paradigm abstract interface, IDataFlow, use a parameterised type. This makes sense because only the application, the Thermometer, knows the actual types it needs to use.

Abstractions and Instances

All software architectures should contain two concepts for its elements equivalent to abstractions and instances.

Abstractions are design-time elements. Instances are run-time elements. Object oriented programming has the two concepts in classes and objects. But many discussions on software architecture seem to combine them into one term, such as modules, components or layers. They may implicitly contain the separate concepts, as components may, but not having them explicit will inevitably lead to confusion.

The problem is their different dependencies. Dependencies between Instances are run-time dependencies. Dependencies between Abstractions are knowledge dependencies. If we don’t have separate terms for design-time and run-time elements, we will tend to implement run-time dependencies in the design-time elements, destroying them as abstractions.

Nearly all common layering schemes have this problem. A common example of the problem is associations between classes. The most important idea that OOP brought us, the idea of different design-time elements and run-time elements, has been ruined by associations. They encourage you to implement run-time dependencies between classes, an anti-pattern in ALA.

What we should be doing is representing the knowledge of run-time dependencies between instances inside another abstraction. By using the terms Abstraction and Instance, ALA honours the separation of run-time elements and design-time elements, and reminds us not to implement runtime dependences between abstractions. at design-time they are not dependencies at all.

6.5. Composition versus decomposition

Here we revisit the important idea first introduced in section 2.4 to do with the pitfalls of thinking in terms of hierarchical decomposition. Each level decomposes the system into smaller elements or components with relations between them. The process continues until the pieces are simple enough to understand.

Decomposition of the system into elements and their interactions.

The decomposition approach is often the de facto or informal method used by developers because it is encouraged by many architecture styles and patterns, for example components or MVC. It is the method used in ADD (Attribute Driven Design). Indeed some definitions of software architecture sound like this meme:

  • From Wikipedia quoting from Clements, Paul; Felix Bachmann; Len Bass; David Garlan; James Ivers; Reed Little; Paulo Merson; Robert Nord; Judith Stafford (2010:

    "Each structure comprises software elements, relations among them, and properties of both elements and relations."
  • IBM.com

    "Architecture is the fundamental organization of a system embodied in its components, their relationships to each other, and to the environment, and the principles guiding its design and evolution. [IEEE 1471]
  • synopsys.com

    "Architecture also focuses on how the elements and components within a system interact with one another."
  • From an article on coupling by Martin Fowler https://www.martinfowler.com/ieeeSoftware/coupling.pdf

    "You can break a program into modules, but these modules will need to communicate in some way—otherwise, you’d just have multiple programs."
  • Loose coupling and high cohesion

This last one suggests that loose coupling is the best we can do. Modules or components must collaborate in some way. It all sounds reasonable, even self-evident. So why is it completely wrong?

To be fair, some of the examples above are vague enough to be interpreted in the right way. But all are suggestive of the idea of decomposition.

To fix the problem, we should re-word the meme:

Expression of the requirements by composition of abstractions.

All four big words are changed and some are exact opposites. Indeed, the architecture that comes out of this method is "inside out" when compared to the decomposition method.

Let’s contrast two pseudo-structures: one that results from the decomposition approach and one that results from the composition approach.

6.5.1. Decomposition of the system into elements and their interactions

This diagram shows a decomposition structure. The outer box is the system. It shows decomposition into four elements, and then those in turn are decomposed into four elements each.

Decomposition structure
Figure 24. Decomposition Structure

The outer elements correctly only refer to the outer interface of the components - their package or namespace interface, facade, or aggregate root - however you want to think of it. Encapsulation is used at every level of the structure to hide implementation details.

The elements are labelled with numbers to emphasise that they are not good abstractions. Of course, in practice these elements are abstract to a point (they have a name), but abstraction is relative.

The next diagram shows the same structure but with parts relevant to a user story marked in red. This is the "their interactions" part of the "The decomposition of your system into elements and their interactions".

Decomposition structure
Figure 25. Tracing a User story

The diagram shows both composition relationships (boxes inside boxes) and interaction relationships (lines).

6.5.2. Expression of the requirements by composition of abstractions

This diagram shows a composition structure.

Decomposition structure
Figure 26. Composition Structure

Only 'composition' relationships are present. We have shown some of them as lines even though you wouldn’t normally draw them. For example, one of many is shown from the top layer to the middle layer, the one from c to C. In practice we wouldn’t normally draw a diagram like this at all - the abstractions would be just referred to by name. But here we are trying to make a combined diagram of the meta-architecture and the specific architecture. The meta-architecture is the three layers, and the knowledge dependencies that go from the higher layers to the lower layers. The specific architecture consists of the diagrams inside the user stories in the top layer, the specific composition of instances.

Note that although we use lines in the diagrams in the top layer, those lines do not represent dependencies.

6.5.3. Comparison of the two approaches

Table 1. Comparison of two approaches
Decomposition Composition

The maintenance cost (effort per user story or effort per change) increases over time. This is because complexity is increasing. Changes will tend to have ripple effects, but that isn’t the biggest problem. Even if a change ends up being in one place, reasoning about the system to determine where that change should be can require reasoning across the system.

The maintenance cost reduces as the system grows. This is because as the domain abstractions mature, the user stories become less and less work to do - they simply compose, configure and wire together instances of existing domain abstractions.

Decomposition structure

Decomposition structure

Hierarchical (fractal) structure

Layered structure

Elements become less abstract as you zoom in. They are specific parts of specific parts.

Parts become more abstract as you go down.

Hides details through encapsulation, which works at compile-time.

Hides details through abstraction, which works at design-time.

Encapsulated parts are made private within increasingly small scopes. These private parts still need to be know about at design-time to understand the system (unless they are also good abstractions).

Lower parts are made public in increasingly large scopes. Only the abstractions themselves are needed to understand the system.

Dependencies go in the direction from the outermost element to the innermost. This is the direction of the less abstract and therefore less stable.

Dependencies go down the layers, in the direction of the more abstract, and therefore more stable.

Dependencies also exist between parts at the same hierarchical level

There are no dependencies between abstractions at the same level.

Encourages the same element to be used for both abstraction and instance - often called a module or component.

Clearly has two distinct types of elements - abstractions and instances.

Elements are loosely coupled (bad). Elements that don’t have dependencies will typically still be implicitly coupled (collaboration).

Abstractions are zero coupled. (The content of any abstraction is zero coupled with the content of any other abstraction).

Discourages reuse. 16 elements all different from each other.

Encourages reuse. Only 5 abstractions. 16 instances of those five abstractions.

SMITA - Structure missing in the action. If you are interested in a particular user story, you will typically have to trace it through multiple elements, multiple interfaces, and their interactions across the structure. An example of this is shown by the diagram with the red lines.

Eliminates this problem. The structure is explicit and in one place.

Coupling increases during maintenance. This is because details are not hidden inside abstractions, only encapsulations. Any of them can be needed at any time by an outer part of the structure. So as maintenance proceeds, more of them will need to be brought into the interfaces, increasing the coupling as time goes on.

Coupling remains at zero during maintenenace. Abstractions represent ideas, and ideas are relatively stable even during maintenance. All the dependencies are relatively unaffected. An operation called generalizing an abstraction is sometimes done. This increases the versatility, reuse and ubiquity of abstractions over time.

Because of the loose coupling, the complexity increases as the system gets larger.

The complexity stays constant as the system gets larger. This is because abstractions are zero coupled. Each abstraction is its own stand-alone program. If we choose an ideal granularity of say 200 lines of code, the complexity in any one part of the program is that of 200 lines of code.

6.5.4. Transforming a decomposition structure into a composition structure

  • The structure turns inside out. Abstractions are found in the inner-most encapsulations. These are brought out to be made public, reusable, ubiquitous and stable at the domain abstractions layer.

  • The parts of the inner encapsulations that are specific to the application are brought out to be configuration information in the application layer that is used to configure instances of the abstractions.

  • Dependencies that existed between encapsulated elements for run-time communications are eliminated. They become simply wiring up of instances.

6.5.5. Smells of decomposition

  • Hierarchical diagrams

The tell-tale sign that this is happening is when we draw hierarchical diagrams. Boxes contained inside boxes. Even if we don’t draw them that way, the 'containment' or encapsulation is still implied. This is what package and componenet diagrams do. ALA has no use for package diagrams in the logical view. (However, they are still relevant in other views. There are several good reasons to have separately deployable binary code units such as exes or dlls.)

  • The dependency graph has many levels

If you have avoided circular dependencies, your application can be viewed as a (compile-time) dependency graph. Because it has run-time dependencies, it will have many 'levels'. In ALA, the dependency graph has just three (or at most four) layers.

  • Encapsulation without abstraction

Encapsulating details without an abstraction causes module or component boundaries to look relatively transparent at design-time. Their so called interfaces will tend to be increasingly wide as the software life cycle proceeds.

  • Modules have responsibility for who they communicate with

Either the sender knows who to send messages to, or, if using publish/subscribe, the receiver knows who to receive messages from. Understanding the system requires reading inside the parts to get the interconnection knowledge.

  • Indirection is making the structure harder to see.

There is a meme floating around that using indirection inevitably makes the structure harder to understand. At first this seems reasonable. At the point of the indirection you cannot immediately see where the execution flow goes next. It seems that you always need to compromise between explicit structure and loose coupling. However the meme is completely false.

In ALA, there is no conflict between zero coupling through indirections and explicitly wired structure.

When the code you are reading is an abstraction, you don’t need to know where the execution flow goes. The abstraction ends at its output ports. To get the big picture, you want to see all the instances of abstractions wired together (they have a coherent purpose). Here you can see the explicit structure, with no indirection and yet zero coupling.

6.6. Expression of requirements

One of the fundamental aspects of ALA is that the abstraction level of the application is fixed and defined by:

The succinct description of requirements

This is a similar concept to a DSL (but not quite the same). If the abstraction level were more specific, we wouldn’t have the versatility to describe changing requirements or new applications in the domain (too expressive). If it were were more general, we would have to write more code to describe the requirements (not expressive enough).

I noticed during 40 years of code bases written at our company, two did not deteriorate under maintenance. They always remained as easy to maintain as they were in the beginning, if not easier. All others deteriorated badly. Some deteriorated so badly that they could no longer be maintained at all. At the time we din’t know why and could not predict which way it would go. It seemed as if you just got lucky or unlucky.

Perhaps it was the type of changes that came along? But the two code bases that were easy to maintain seemed to be easy for any kinds of change. And the ones that were hard were hard for any change. This continued to hold for years on end. Of course, most changes were changes to requirements, but often enough, changes would be for performance or other reasons. These also seemed easy in these two code bases, but hard everywhere else.

I began to look at the structure and style of the easy and hard code. The easy code was not complicated while the hard code had degenerated well into the complex. The two easy code bases were doing very different things in very different ways, so there was apparently not a common structure or style. But they did have one thing in common. The code that represented the knowledge of the requirements was separated out. That code only described requirements, and it was expressed in terms of other things that were relatively independent, reusable and easy to understand (what we call abstractions).

This is what first gave rise to one of the core tenets in ALA. The first separation is not along the lines of functional or physical parts of the system, such as UI, Business logic, and Data model. The first separation is code that just describes requirements.

Of course this has a strong parallel with how DSLs work. Is ALA just DSLs? There are several differences. Firstly in ALA we don’t try to create a sandbox language for a domain expert to maintain applications. We don’t go as far as an external DSL. It’s for the developer and we don’t want to cut him off from the power he already has when it is needed. We just give him a way to organise the code and a process to get him there - describe the requirements knowledge in terms of abstractions and then trust that those abstractions, when written, will make it work.

6.7. No two modules know the meaning of data or a message.

The two modules will have collaborative knowledge. We reason that the sender must know the meaning to formulate the message, and the receiver must know the meaning to interpret the message. So how can it be avoided? The answer is to make the sender and receiver in same abstraction. They both know the same knowledge, so they are cohesive, so they should be together. In the logical view of the system, they are two instances of the one abstraction. We let the physical view fact that the sender and receiver will be deployed in different places drive them to be different modules.

6.8. Expressiveness

Requirements are usually understated initially in terms of abnormal conditions. However, they are usually communicated quite quickly relative to the time to write the code. In ALA, they are separately represented. The precise expression of the requirements using the right programming paradigms should take about the same amount of information as the English explanation of them.

In general, ALA probably requires about the same amount of total code. But once the requirements are represented, the domain abstractions are known and they are independent small programs with dependencies only on the programming paradigm interfaces used. This independence should make them much easier to write. As the system matures, the effort to modify gets less as more domain abstractions come on line as tested, mature and useful building blocks. The final cost of maintenance should be much less than an equivalent ball of mud architecture.

6.9. No models

Leave out details only inside abstractions

It is generally accepted that a software architecture must, by necessity, leave out some details. Somehow we need to find a satisfactory architecture without considering all the details. Often models are used to represent the architecture. Like its metaphor in the real world, a model leaves out details. The problem is they can leave out arbitrary details. We can’t be sure that some omitted detail won’t turn out to be important to the architectural design.

ALA therefore does not use the model metaphor. Instead, it uses diagrams (if not plain old text). Of course, this distinction comes down to semantics. I define a diagram as different from a model in that it does not leave out details arbitrarily. The only way to leave out details in an ALA diagram is inside the boxes, in other words inside abstractions. Because abstractions already have the required meaning when used in the diagram, the details omitted can’t be important to the diagram, and can’t affect the architectural design.

6.9.1. Executable architecture

Your architecture should be executable

The distinction between diagrams and models explained in the previous section gives rise to an interesting property of the ALA architecture. Diagrams are executable. Therefore the architecture itself will be executable. When the implementation of the abstractions is complete, there will be no work left to do to make the architecture execute (apart from practical considerations of bugs, misinterpretations of the requirements, performance issues, improvements to the initially conceived set of domain abstractions, and the like).

There should be two aspects of an architecture, the meta-architecture and the specific architecture. If using ALA, ALA itself is the meta-architecture and the top level application diagram is the specific architecture.

If your specific architecture is executable, it is also code. There is no separate documentation or model trying to act as a second source of truth.

6.9.2. Granularity

The final architecture of your software will consist only of abstractions. These abstractions will need to be independently readable and understandable. To meet this need, all of the abstractions will be small, even the 'system level' ones.

Conversely, none should be too small. We want them small enough to allow the human brain to understand them, but there is no need for them to be smaller, or we will just end up with an inordinate number of them. This inordinate number will tax the brain in a different way, by causing it to have to learn more abstractions than necessary in a given domain.

The ideal abstraction size is probably in the range of 50 to 500 lines of code.

6.9.3. Modules, Components, Layers

The common terms, modules, components, or layers often result from a decomposition process and therefore are parts of a specific system. The system may have only one of each type. The parts have a lower abstraction level than the system because they are just specific parts of it. In ALA we want to reverse this so that parts are more abstract than the system.

But say you do end up with some single use abstractions and implement it in a static way, it is important to still see these entities as two aspects in one: an abstraction and an instance.

6.10. Abstraction Layers

6.10.1. Layers pattern

With only design-time knowledge dependencies to deal with, layers are used for organising these dependencies so that there are no circular dependencies, and that they all go toward more abstract, more stable abstractions. As the name "Abstraction Layered Architecture" suggests, layers are crucially important to ALA.

In the section on the wiring pattern we ended with three layers:

diagram 21.png

There is a Layers pattern that also controls dependencies, but since most systems have numerous run-time dependencies between elements represented as design-time dependencies, these layers are used for the run-time dependencies. It is usually explained that each layer is built on services provided by the layer below it.

One example is the UI/Business Logic/Data model. Another example is the OSI communications model, where the layers are Application, Presentation, Session, Transport, Network, Data link, and Physical. In ALA, each of these ends up being turned 90 degrees. Metaphorically they become chains. In ALA each component wouldn’t know about the components next to it. That applies symmetrically, to the left and to the right. Data goes in both directions. At run-time, everything must exist for the system to work. It doesn’t really make sense to use a asymmetrical layers metaphor.

The design pattern for layers does have one or two examples of layering used by knowledge dependencies. The term ‘layer’ is therefore an overloaded term in software engineering. When used for knowledge dependencies, the English term 'layer' is a better metaphor. If a lower layer of a wall were to be removed, the layers above would literally collapse, and that’s exactly what would happen in knowledge dependency layering. The layers above literally need the knowledge of abstractions in lower layers to make any sense.

ALA’s ripple effects are already under control because the only dependencies are on abstractions, which are inherently stable, and furthermore, those abstraction must be more abstract. However, to make these dependencies even less likely to cause an issue during maintenance, we try to make the abstraction layers discrete, and separated by approximately an order of magnitude. In other words each layer is approximately an order of magnitude more abstract than the one above it. More abstract means more ubiquitous, so the layers contain abstractions which have greater scope, and greater potential reuse as you go down the layers.

We won’t need many layers. If you think about abstraction layers in the real world, we can get from atoms to the human brain in four layers. Remember the creativity cycle early in this article. We only need to go around the cycle four times to make a brain: Atoms, Molecules such as proteins, Cells such as neurons, neural nets, and finally the brain itself.

6.10.2. The four layers

We start with four layers. They have increasing scope as you go down. This type of layering was described by Meiler Page-Jones. Meiler Page-Jones’ names for the four layers are: "Application domain", "Business domain", "Architecture domain", and "Foundation domain".

Layers diagram
Figure 27. Four ALA layers

ALA uses slightly different names: Application layer, Domain Abstractions layer, Programming Paradigms layer, and Language layer.

Application layer

The top layer has knowledge specific to the application, and nothing but knowledge specific to the application, i.e. representing your requirements.

A simple Application might wire a grid directly to a table. When Business logic is needed, any number of decorators (that do validation, constraints, calculations, filtering, sorting, etc.) can be inserted in between the grid and the table by changing the wiring of the application.

Domain abstractions layer

Knowledge specific to the domain goes in this layer. A domain might correspond to a company or a department. As such, teams can collaborate on the set of abstractions to be provided there.

Applications have knowledge dependencies reaching into this layer.

Programming Paradigms layer

All knowledge specific to the types of computing problems you are solving, such as execution models, programming paradigm interfaces and any frameworks to support these, is in this layer.

The Programming Paradigms layer will abstract away how the processor is managed to execute different pieces of code at the right time. Execution models are covered in detail in chapter four.

This layer is also where we arrange for our domain abstractions to have common simple connections instead of having a specific language for each pair of modules that communicate. The Programming Paradigms layer abstracts away ubiquitous communications languages (which we have been referring to as programming paradigms in this article.)

Let’s use the clock as a real world example. (This is the same clock example we used in section 2.9 when introducing the role abstractions play in the creative process.) One of the the domain abstractions for clocks is a cog wheel. Cog wheels communicate with one another. But they don’t do it with communications languages specific to each pair, even though each pair must have the correct diameters and tooth sizes to mesh correctly. The cog abstraction just knows about the common paradigm of meshing teeth, a more abstract language in this lower layer. This language is analogous to a programming paradigm. With it, the clock abstraction (which is in the highest layer) can then instantiate two cogs and configure them to mesh. The concept of cog thus remains an abstraction and instances of it are composable. The clock, which already knows that two instances of cogs are needed, also knows where they will be fitted and what their diameters must be. The knowledge in the clock abstraction is cohesive.

Language layer

The language layer is included to show what is below the other three layers. It is not hardware as you would find in many other layering schemes, nor is it a database, because it is not run-time dependencies we are layering. The lowest layer has the remaining knowledge you need to understand your code, that of the languages, libraries and any very generic APIs you may use.

The hardware and database do have a place, but we will cover it later. Being a run-time dependency, it will be well off to one side and slightly higher up.

Domain Abstractions API

The boundary between the application layer and the domain abstractions layer is an API that supports the solution space of your requirements (within the constraints of your domain).

The scope of the Domain Abstractions layer defines the expressiveness available to the application. The greater the scope (or bigger the domain), the more applications are able to do. The cost is expressiveness. The applications will have to be longer to specify what is to be done. Conversely, a smaller domain allows less versatility in the applications, but there is greater expressiveness, which means you write less code.

Possible extra layers

The domain is an approximation of all the potential applications and all the modifications you are likely to make. If the domain is large because it is enterprise wide, you could have an additional layer for small domains. The enterprise domain would include enterprise wide abstractions such as a person identity, and the smaller domains would add additional, more specific abstractions, such as a customer (by composition).

If the applications are large and themselves need to be composed of features, an additional layer that supports plug-in style abstractions may work well. Plug-in abstractions may actually be instances of domain abstractions, such as a settings Menu, or a customer Table. A feature can then add settings to the menu, or columns to the table that remain unknown to any other features.

Programming Paradigms API

The boundary between all higher layers and the Programming Paradigms layer is another API. It separates the domain knowledge from the programming paradigm implementation knowledge. It almost always takes care of the ‘execution flow’, the way the computer CPU itself will be controlled to execute all the various parts of the code and when, often using a framework. On the other hand, the Programming Paradigms layer doesn’t necessarily have any code at all. Remember that the layers are ‘knowledge dependencies’, not run-time dependencies, so the paradigm could be a ‘computational model’ that just provides the knowledge of patterns of how to constracut the code in higher layers. The decisions about use of the patterns and about the way the code is executed have already been made and exist in the Programming Paradigms layer.

Rate of change of knowledge

The knowledge in each of the four layers has different change rates.

  • The Language layer contains knowledge that will likely change only a few times in your career.

  • The Programming Paradigms layer knowledge changes when you move to different computing problems types, or discover different approaches to solving a broad range of problems. For example, if you have not yet used an event driven execution model or state machines in your career, and you move into the embedded systems space, you will very likely need to have those skills.

  • The Domain Abstractions layer has knowledge that changes when you change the company you work for. It will change at the rate that the company’s domain is changing, or is becoming better understood. If your company uses lean principles, one of the things you want to do is capture knowledge for reuse. This is the whole point of the Domain Abstractions layer, it is a set of artefacts that capture the company’s reusable knowledge.

  • The Application layer has the fastest changing knowledge, the knowledge that changes at the rate that an application gets maintained.

6.11. ALA is a logical view

If the system is deployed on multiple machines (this is the subject of the physical view), the ALA abstractions, layers and diagrams all remain identical. A simple application diagram connecting a temperature sensor to a display field does not change if the sensor happens to be on a Mars Rover and the display field is at JPL.

Ideally, the performance view also does not affect the ALA logical view. This is a many faceted problem that we will return to later.

ALA usually works very well with aspects of the development view as discussed elsewhere. For example, the fact that domain abstractions have zero coupling greatly helps the allocation of teams. The teams need only cooperate on a common understanding of the programming paradigms used.

6.12. No separation of UI

In ALA we don’t separate the UI unless there is a reason to do so. The amount of knowledge in the UI that comes from a particular application’s requirements is usually quite small and that knowledge is usually quite cohesive and coupled with the business logic of the feature it belongs with. For example, the layout of the UI is a small amount of information, and the bindings of the UI elements to data are a small amount of information. So all that cohesive knowledge is kept together, encapsulated inside a feature. Instead, the UI is composed from Domain UI abstractions. Being domain specific, these abstractions have a little more knowledge to them than generic widgets. For example, their domain knowledge may include style, functionality and suitability to their domain context. For example, a softkey or menu item will have an appearance, functionality and suitability to the way UIs are designed in the domain. Using one in a specific application only requires a label and a binding to an action. They will also provide consistency in the domain.

If there is an actual requirement to have different UIs, say a command based UI and a GUI based UI, then you just abstract the UI abstractions further until they can be used either way. The UI abstractions still remain an integral part of the application.

In the example project for this chapter, we will for the first time use multiple programming paradigms, a usual thing in real ALA projects.

6.13. Features

You may have noticed throughout this article the word 'features' being used quite often instead of 'Application'. When the application is large, we can think of it as a composition of feature abstractions. This is exactly what happens in natural language in the domain when describing requirements. 'Features' is just the word we give the natural abstractions in the requirements, without even realizing it. Just go with this in the software itself.

6.14. Horizontal domain partitions

Say you are implementing a particularly large domain abstraction such as a 'Table', or are implementing a complicated programming paradigm. We would like to break these up into smaller components. Do we introduce a fractal type of structure to deal with this? Should we have hierarchical layers within layers contained completely inside the Table abstraction?

The astute reader will have noticed the non-ALA thinking in the statement "break these up into smaller components". In ALA we don’t decompose a large abstraction into components, we compose it from abstractions, which if necessary we invent as we go. These new abstractions will have a scope or level of ubiquity, stability and reuse that corresponds to one of the existing layers. So there should be no hierarchical or fractal structures in ALA.

However, the domain that these new abstractions are in won’t be the same domain as the one that provides for the writing of Application requirements. For example, the implementation of the Table abstraction will need to be connected to another abstraction in the domain of databases. One of the abstractions in that domain will know about a particular database, say SQL Lite. A polymorphic interface should exist between the two. That interface, being more abstract than either the Table or the SQL Lite abstractions, will be in the next layer down, where both the Table and the MySQL abstractions can have a knowledge dependency on it. Of course the SQL abstraction will actually be further composed of an adapter and a real database.

Some application domain abstractions are complicated. Examples of these are abstractions requiring a connection to an actual database, actual hardware, the Internet, etc. Implementing these will typically wire out horizontally into other technical domains. You can visualise them going in multiple directions, which is exactly the idea of Alistair Cockburn’s hexagonal architecture.

diagram 22.png

A communications domain using a OSI model may end up with a whole chain of communications domain abstractions going sideways:

diagram 23.png

The technicalities may be incorrect but the diagram gives the idea of how the OSI 'layers', which are just run-time dependencies, would fit into the ALA layers.

6.15. No hierarchical design

ALA does not use any form of hierarchical structure. Instead it uses abstraction layers, together with "Horizontal domain partitions" discussed earlier.

6.16. Product owner perspective

TBD

6.17. Reuse

TBD

6.18. Documentation

TBD

6.19. Symbolic indirection

Avoid use of symbolic indirection without abstraction

When we start assembling requirements from abstractions, a topic that we will cover in coming sections, we will be using symbolic indirection, such as function calls or the new keyword with a class name. Unless a symbolic indirection is to an abstraction, they are for the compiler to follow at compile-time, not for the code reader to follow at design-time. Understanding the code relies on allowing the reader to read a small cohesive block of code. The reader should never have to follow the indirection somewhere else. If you don’t achieve this, and abstraction is the only way you can, then any decoupled architecture will be more difficult to read.

Abstraction allows indirection while allowing the reader to continue reading on to the next line. The importance of this property cannot be overstated. As soon as we start thinking in mere programming language terms of modules, components, interfaces, classes, or functions, the abstraction will start to be lost. These other artefacts may have benefits at compile-time (the compiler can understand them), but that is not useful at design-time unless they are also good abstractions.

It would be nice if your compiler could tell you that you have a missing abstraction, just as it does for a missing semicolon, but alas, they are not capable of understanding abstractions yet. So it is still entirely up to you.

Abstraction is almost a black and white type of property. It’s either there or it isn’t. If the reader of your code does not have to follow the indirection, you have it.

Footnote: When the reader of your code meets your abstraction for the first time (usually a domain abstraction in a domain they have recently come into), ideally their IDE will give them the meaning in a little pop-up paragraph as their mouse hovers over any of its uses. Depending on the quality of the abstraction, after a single exposure, their brain will have the insight, like a light coming on, illuminating a meaning. The brain will form a new neuron to represent the concept. Since the reader will hopefully remain in the domain for some time, this overhead to readability shouldn’t be large.

6.20. Everything through interfaces

A class, in contrast to an abstraction, has an interface comprising all the public members. In ALA we only want this interface to be used by the application when it instantiates and configures an instance of an abstraction. All other inputs and outputs that are used at run-time are done through interfaces (abstract interfaces).

6.21. What do you know about?

Whenever I have only two minutes to give advice on software architecture, I use this quick tip. The tip is ALA reduced to its most basic driving principle.

Ask your modules, classes and functions:

What do you know about?

The answer should always be "I just know about…​".

The anthropomorphization helps the brain to see them as abstractions. The word 'knows' is carefully chosen to imply a 'design-time' perspective.

  1. It’s a restatement of the SRP (Single Responsibility Principle). Every element should know about one thing, one coherent thing. Furthermore, no other elements should know about this one thing.

  2. An element may know about a single hardware device.

  3. An element may know about a user story.

  4. An element may know about a protocol.

  5. An element may know an algorithm.

  6. An element may know how to do an operation on some data, or the meaning of some data, but not both.

  7. An element may know a composition of other elements.

  8. An element may know where data flows between other elements.

  9. No element should know the source or destination of its inputs and outputs.

6.22. Example project - Game scoreboard

For the example project for this chapter, we return to the ten-pin bowling and tennis scoring engines that we used in Chapter two, and add a scoreboard feature (well a simple ASCII scoreboard in a console application rather than real hardware).

As the requirement, say we want a console application that displays ASCII scoreboards that look like these examples:

Ten-pin

 -----+-----+-----+-----+-----+-----+-----+-----+-----+--------
|   1 |   2 |   3 |   4 |   5 |   6 |   7 |   8 |   9 |    10  |
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
| 1| 4| 4| 5| 6| /| 5| /|  | X| -| 1| 7| /| 6| /|  | X| 2| /| 6|
+  +--+  +--+  +--+  +--+  +--+  +--+  +--+  +--+  +--+  +--+--+
|   5 |  14 |  29 |  49 |  60 |  61 |  77 |  97 | 117 |   133  |
 -----+-----+-----+-----+-----+-----+-----+-----+-----+--------
Tennis

 -----++----+----+----+----+----++--------
|   1 ||  4 |  6 |  5 |    |    ||    30  |
|   2 ||  6 |  4 |  7 |    |    ||  love  |
 -----++----+----+----+----+----++--------

As usual in ALA, our methodology begins with expressing those requirements directly, and inventing abstractions to do so. So, we invent a 'Scorecard' abstraction. It will take a configuration which is an ASCII template. Here are the ascii templates that would be used for ten-pin and tennis:

 -------+-------+-------+-------+-------+-------+-------+-------+-------+-----------
|   1   |   2   |   3   |   4   |   5   |   6   |   7   |   8   |   9   |     10    |
+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+
|F00|F01|F10|F11|F20|F21|F30|F31|F40|F41|F50|F51|F60|F61|F70|F71|F80|F81|F90|F91|F92|
+   +---+   +---+   +---+   +---+   +---+   +---+   +---+   +---+   +---+   +---+---+
|  T0-  |  T1-  |  T2-  |  T3-  |  T4-  |  T5-  |  T6-  |  T7-  |  T8-  |    T9-    |
 -------+-------+-------+-------+-------+-------+-------+-------+-------+-------------
 -----++----+----+----+----+----++--------
| M0  ||S00 |S10 |S20 |S30 |S40 || G0---  |
| M1  ||S01 |S11 |S21 |S31 |S41 || G1---  |
 -----++----+----+----+----+----++--------

The scorecard ASCII template has letter place-holders for the scores. (A single letter is used so it doesn’t take up much space on the template design.) Different letters are used for different types of scores. Digits are used to specify where multiple scores of the same type are arranged on the scoreboard. They are like indexes. Either 1-dimensional or 2-dimensional indexes can be used in the scoreboard template. For example, the frame scores in ten-pin bowling have scores for each ball for each frame, F00, F01 etc, as shown in the example above.

The scorecard abstraction needs functions it can use to get the actual scores. The functions are configured into little 'binding' objects that we then wire to the scoreboard. The binding objects are configured with the letter that they return the score for.

6.22.1. Ten-pin

Having invented the Scorecard and Binding abstractions, we can now do the ten-pin application diagram:

diagram bowling 3.png

An abstraction we didn’t mention yet is the ConsoleGameRunner. Its job is to prompt for a score from each play, display the ASCII scoreboard, and repeat until the game completes.

The 'game' instance of the Frame abstraction on the right of the diagrams is the scoring engine we developed in Chapter Two. Together with this engine, we now have a complete application.

The rounded boxes in the diagram are instances of domain abstractions as usual for ALA diagrams. The sharp corner boxes are instances of Application layer abstractions. They are the mentioned functions for the Bindings. That code is application specific so goes in the application layer. They just do a simple query on the scoring engine.

Now tranlate the diagram into code. Here is the entire application layer code for ten-pin:

consolerunner = new ConsoleGameRunner("Enter number pins:", (pins, engine) => engine.Ball(0, pins))
.WireTo(game)
.WireTo(new Scorecard(
"-------------------------------------------------------------------------------------\n" +
"|F00|F01|F10|F11|F20|F21|F30|F31|F40|F41|F50|F51|F60|F61|F70|F71|F80|F81|F90|F91|F92|\n" +
"|    ---+    ---+    ---+    ---+    ---+    ---+    ---+    ---+    ---+    ---+----\n" +
"|  T0-  |  T1-  |  T2-  |  T3-  |  T4-  |  T5-  |  T6-  |  T7-  |  T8-  |    T9-    |\n" +
"-------------------------------------------------------------------------------------\n")
.WireTo(new ScoreBinding<List<List<string>>>("F",
    () => TranslateFrameScores(
        game.GetSubFrames().Select(f => f.GetSubFrames().Select(b => b.GetScore()[0]).ToList()).ToList())))
.WireTo(new ScoreBinding<List<int>>("T",
    () => game.GetSubFrames().Select(sf => sf.GetScore()[0]).Accumulate().ToList()))
);

If you compare this code with the diagram, you will see a pretty direct correspondence. Remember 'game' is the reference to the scoring engine project in the previous chapter.

That’s pretty much all the code in the application. Oh there is the 'translate' function, but it is pretty straight forward once you know the way a ten-pin scorecard works. For completeness here it is.

/// <summary>
/// Translate a ten-pin frame score such as 0,10 to X, / and - e.g. "-","X".
/// </summary>
/// <example>
/// 7,2 -> "7","2"
/// 7,0 -> "7","-"
/// -,3 -> "-","7"
/// 7,3 -> "7","/"
/// 10,0 -> "",X
/// 0,10 -> "-","/"
/// additional ninth frame translations:
/// 10,0 -> "X","-"
/// 7,3,2 -> "7","/","2"
/// 10,7,3 -> "X","7","/"
/// 0,10,10 -> "-","/","X"
/// 10,10,10 -> "X","X","X"
/// </example>
/// <param name="frames">
/// The parameter, frames, is a list of frames, each with a list of integers between 0 and 10 for the numbers of pins.
/// </param>
/// <returns>
/// return value will be exactly the same structure as the parameter but with strings instead of ints
/// </returns>
/// <remarks>
/// This function is an abstraction  (does not refer to local variables or have side effects)
/// </remarks>
private List<List<string>> TranslateFrameScores(List<List<int>> frames)
{
    // This function looks a bit daunting but actually it just methodically makes the above example tranlations of the frame pin scores
    List<List<string>> rv = new List<List<string>>();
    int frameNumber = 0;
    foreach (List<int> frame in frames)
    {
        var frameScoring = new List<string>();
        if (frame.Count > 0)
        {
            // The first 9 frames position the X in the second box on a real scorecard - handle this case separately
            if (frameNumber<9 && frame[0] == 10)
            {
                frameScoring.Add("");
                frameScoring.Add("X");
            }
            else
            {
                int ballNumber = 0;
                foreach (int pins in frame)
                {
                    if (pins == 0)
                    {
                        frameScoring.Add("-");
                    }
                    else
                    if (ballNumber>0 && frame[ballNumber]+frame[ballNumber-1] == 10)
                    {
                        frameScoring.Add(@"/");
                    }
                    else
                    if (pins == 10)
                    {
                        frameScoring.Add("X");
                    }
                    else
                    {
                        frameScoring.Add(pins.ToString());
                    }
                    ballNumber++;
                }

            }
        }
        rv.Add(frameScoring);
        frameNumber++;
    }
    return rv;
}

6.22.2. Tennis

So now that we have these domain abstractions for doing console game scoring applications, lets do tennis:

diagram tennis 3.png

I left the code out of the GetGameOrTieBreakScore box as it is a little big for the diagram here. It is similar to the other queries but it must first determine if a tie break is in progress and get that if so. Also it translates game scores from like 1,0 to "15","love".

And here is the code for the Tennis diagram:

consolerunner = new ConsoleGameRunner("Enter winner 0 or 1", (winner, engine) => engine.Ball(winner, 1))
.WireTo(match)
.WireTo(new Scorecard(
        "--------------------------------------------\n" +
        "| M0  |S00|S10|S20|S30|S40|S50|S60|  G0--- |\n" +
        "| M1  |S01|S11|S21|S31|S41|S51|S61|  G1--- |\n" +
        "--------------------------------------------\n")
    .WireTo(new ScoreBinding<int[]>("M", () => match.GetScore()))
    .WireTo(new ScoreBinding<List<int[]>>("S", () =>
        match.GetSubFrames()
            .Select(sf => sf.GetSubFrames().First())
            .Select(s => s.GetScore())
            .ToList())
    .WireTo(new ScoreBinding<string[]>("G", () => GetGameOrTiebreakScore(match)))
);

If you compare this code with the diagram, you can see a pretty direct correspondence. match comes from the scoring engine project in Chapter two.

6.22.3. Concluding notes

Although the diagrams must be turned into text code to actually execute, it is important in ALA to do these architecture design diagrams first. They not only give you the application, they give you the architectural design by giving you the domain abstractions and programming paradigms as well. If you try to design an ALA structure in your head while you write it directly in code, you will get terribly confused and make a mess. Using UML class diagrams will make it even worse. Code at different abstraction levels will end up everywhere, and run-time dependencies will abound. Our programming languages, and the UML Class diagram, are just not designed to support abstraction layered thinking - it is too easy to add bad dependencies (function calls or 'new' keywords) into code in the wrong places.

Note that at run-time, not all data-flows have to go directly between wired up instances of domain abstractions. The data can come up into the application layer code, and then back down. This was the case when we did the functional composition example in Chapter One. In this application we are doing that with the code in the square boxes that get the score from the engine. The important thing is that all the code in the application is specific to the application requirements.

That completes our discussion of the console applications for ten-pin and tennis. The full project code can be viewed or downloaded here:

7. Chapter seven - ALA compared with…​

7.1. Physical boundaries

I was listening to a talk by Eric Evans where he said that Microservices works because it provides boundaries that are harder to cross. We have been trying to build logical boundaries for 60 years, he said, and failed. So now we use tools like Docker that force us to use say REST style interfaces in oder to have physical boundaries. I have also heard it suggested that using multiple MCUs in an embedded system is a good thing because it provides physical boundaries for our software components. And I think, really? Is that the only way we can be create a logical boundary? I can tell you that multiple MCUs for this reason is not a good idea if only because all those MCUs will need updating, and the mechanisms and infrastructure needed to do that make it not worth it. Unless there is a good reason, such as to make different parts of your code independently deployable, the extra infrastructure required for physical boundaries that are just logical boundaries is not necessary. Furthermore, physical boundaries, like modules do not necessarily make good abstractions. The only boundary that works at design-time is a good abstraction. So ALA achieves it’s design-time boundaries by using abstractions.

7.2. Test Driven Development

It is said that TDD’s main advantage is not so much the testing, but the improvement in the design. In other words, making modules independently testable makes better abstractions. This is probably true, but in my experience, TDD doesn’t create good abstractions nearly as well as pursuing that goal directly. The architecture resulting from TDD is better but still not great.

7.3. Observer pattern and dependency inversion

TBD

7.4. Hexagonal Architecture (ports and adapters)

In the previous section we intimated that the sideways chains of interfaces going out in horizontal directions were the same as hexagonal architecture. While ALA shares this aspect of hexagonal architecture, there is still an important difference.

ALA retains domain abstractions of the UI, Database, communication and so on. For instance, in our XR5000 example, we had a domain abstraction for a persistent Table. We had domain abstractions for UI elements such as Page, Softkey etc. We don’t just have a port to the persistence adapter, we have an abstraction of persistence. We don’t just have a port for the UI to bind to, we have abstractions of the UI elements. The implementation of these abstractions will then use ports to connect to these external system components. Why is it important that we have domain abstractions of these external components?

  1. The Database and the UI will have a lot of application specific knowledge given them as configuration. Remember the creativity cycle. After instantiation of an abstraction comes configuration. The database will need a schema, and the knowledge for that schema is in the application. The Softkey UI elements will need labels, and that knowledge is in the application. By making domain abstractions for persistence and UI, the application can configure them like any other domain abstraction as it instantiates and wires up the application. To the application, these particular domain abstractions look like wrappers of the actual database and UI implementations, but they are more like proxies in that they just pass on the work.

    The Persistence abstraction then passes this configuration information, via the port interface to the actual database. The Softkey abstraction then passes its label, via the port interface, to the softkeys. Otherwise the Application would have to know about actual databases and actual softkeys.

    If you need a design where the UI can change, you just make the UI domain abstractions more abstract. A softkey may be a command abstraction. It is still configured with a label. But it may be connected to a softkey, a menu item, a CLI command, a web page button, or a Web API command.

  2. From the point of view of a DSL, it makes sense to have concepts of UI and persistence and communications in the DSL language. The application is cohesive knowledge of requirements. The UI and the need for persistence are part of the requirements. In fact, for product owners communicating requirements, the UI tends to be their view of requirements. They talk about them in terms of the UI. Many of the product owners I have worked with actually design the UI as part of the requirements (with the backing of their managers, who are easily convinced that software engineers can’t design UIs. PO can’t either, but that is another story.). The point here is that the UI layout, navigation, and connection to business logic is all highly cohesive. We explicitly do not want to separate that knowledge.

    As a restatement of an earlier tenet of ALA, it is much better to compose the application with abstractions of Business logic, UI and persistence than to decompose the application into UI, persistence and business logic.

  3. We want the application to have the property of composability. We have previously discussed how that means using programming paradigm interfaces for wiring up domain abstractions. By using domain abstractions to represent external components, the abstractions can implement the paradigm interfaces and then be composable with other domain abstractions. For example, the Table domain abstraction which represents persistence may need to be connected directly to a grid, or to other domain abstractions that map or reduce it. Indeed, the Table abstraction itself can be instantiated multiple times for different tables and be composed to form a schema using a schema programming paradigm interface. I have even had a table instance’s configuration interface wired to a another Table instance. (So its columns can be configured by the user of the application.)

  4. The fourth reason why it is important for the application to not directly have ports for external components of the system is that we don’t want the logical view of the architecture to become just one part of the physical view. If there is a communications port that goes to a different physical machine where there is more application logic, the application’s logical view should not know about that. It may be presented as an annotation on the application (lines) connecting certain instances, but it shouldn’t split the application up. At the application level, the collaboration between parts instantiated on different machines is still cohesive knowledge and belongs inside one place - the application.

7.5. Layer patterns

7.5.1. MVC

TBD

7.5.2. Application, Services, Drivers, Hardware

TBD

7.6. Factory method pattern

Let’s say you have a nice Table domain abstraction that is perfect for the requirements you have in your domain. At run-time, the Table abstraction must be wired via a polymorphic interface to a particular database, and the database must be instantiated. We don’t want the Application, the part that composes domain abstractions, to know anything about all this. We want the Application to be able to just use a Table as if it is a self-contained abstraction.

Similarly, in the partitioned off 'Database' domain, there will usually be some knowledge in the top layer to configure a particular database. This knowledge knows nothing of the Application and is not responsible for creating the Tables for it.

Table knows about the polymorphic interface, but doesn’t know it’s for a connection to a real database. This interface could have a method that the Table instance calls when it is instantiated. Table only knows that it has to call this method. However, there is not yet any provider of this interface in place, so we cannot get the database connected up this way. But we do want to wait until the first Table abstraction is instantiated, otherwise we would not need to spin up a database at all.

Instead, Table knows to instantiate a Factory design pattern object. To the Table, this action it must do is logically just part of the same polymorphic interface. It doesn’t even know it is a factory. If the interface is called 'IOutsideConnection', the factory could be called something like OutsideConnection and the method inside it called getAnOutsideConnection. It could be a static class or a singleton. Table just saves the object that it gets in the interface reference.

IOutsideConnection outside = (new OutsideConnection()).getAnOutsideConnection();

If it is the first time the factory object is used, it could, for example, invoke an object in the top layer (via dependency inversion, the object has already registered itself with the factory) that contains the configuration knowledge about the database. The object in the top layer instantiates a database domain abstraction to give to the Table.

7.7. Interface segregation principle

TBD

7.8. Open Closed Principle and decorators

TBD

7.9. Bridge pattern

TBD

7.10. Architecture styles

I am not an expert at these so called 'Architectural styles'. Any feedback about the accuracy of the following comparisons would be appreciated.

7.10.1. Presentation, Business, Services, Persistence, Database

TBD

7.10.2. Presentation, Application, Domain, Infrastructure

The middle two layers appear to be the same as ALA’s. The Presentation (UI) only has run-time dependencies on the Application, and the Domain layer only has run-time dependencies on the Infrastructure (Persistence etc), so these layers are not present in ALA.

Instead Presentation is done in the same way as the rest of the application, by composing and configuring abstractions in the domain. The meaning of composition for UI elements (typically layout and navigation-flow) is different from the meaning of composition in the use-cases (typically work-flow or data-flow).

In ALA, the foundation layer is also done in the same way as the rest of the application, at least a little. Domain abstractions that represent say a persistent table are in the Domain layer. The composition and configuration of them again goes in the Application layer. This time the meaning of composition is, for example, columns for the tables and schema relations.

If the implementation of any domain abstraction is not small (as is the case with the persistent Table abstraction mentioned above, which will need to be connected to a real database), it will be using other abstract interfaces (in the Programming Paradigms layer) connected to its runtime support abstractions in a technical domain, the same as in Hexagonal Architecture.

7.10.3. Object Oriented Programming

From my reading, it seems that the most characteristic feature of OOP is that when data and operations are cohesive, they are brought together in an object. Others may see it as enabling reuse, inheritance, and still others may see it as polymorphism. New graduates seem to be introduced to polymorphism in inheritance and not be introduced to interfaces at all, which is a shame because the concept of interfaces is much more important.

I have never been an expert at Object Oriented Design as I found the choice of classes difficult and the resulting designs only mediocre. But I think the most fundamental and important characterising feature of OOP is under-rated. That is the separation of the concepts of classes and objects. This separation is not so clearly marked when we use the terms modules or components. The separation is fundamentally important because it’s what allows us to remove all dependencies except knowledge dependencies. In the way described earlier in this article, you can represent the knowledge of most dependencies as a relationship between instances completely inside another abstraction. What OOP should have done is represent relationships between objects completely inside another class. The problem is that OOP doesn’t take advantage of this opportunity. Instead, it puts these relationships between objects inside those objects' classes, as associations or inheritance, thereby turning them into design-time dependencies, and destroying the abstract qualities of the classes. Abstractions, unlike classes, retain their zero coupling with one another.

ALA addresses the problem by calling classes abstractions and objects instances. Abstractions differ from classes by giving us a way to have logical zero coupling, as if they were on different physical platforms. Instances differ from objects by having ports because their classes give them no fixed relationships with other objects.

Of course, when you are writing ALA code, abstractions are implemented using classes, but you are not allowed associations or inheritance. Instances are implemented as objects but with ports for their connections. A port is a pair of interfaces that allow methods in both directions. The interfaces are defined in a lower layer.

In ALA, the UML class diagram completely loses relevance. Because classes have no relationships with each other, bar knowledge dependencies, a UML diagram in ALA would just be a lot of boxes in free space, like a pallet of things you can use. You could show them in their layers and you could even draw the downward composition relationships that represent the knowledge dependencies, but there would be no point to this except in explaining the concepts of ALA. When you are designing an actual system, the real diagram is the one inside of an abstraction, especially the uppermost one, the application. It shows boxes for instances of the abstractions it uses, with the name of the abstraction in the box, the configuration information for those instances, and of course the lines showing how they are wired together. The names inside the boxes would not even need to be underlined as in UML, because the boxes in such diagrams would always be instances.

Such a diagram is close to a UML object diagram. However, a UML object diagram is meant to be a snapshot of a dynamic system at one point in time. In ALA, any dynamic behaviour is captured in a static way by inventing a new abstraction to describe that dynamic behaviour. Thus the design-time view is always static. So the object diagram is static. The application class specifies a number of objects that must be instantiated, configured, and wired together to execute at run-time. Since the structure is always static, ideally this would be done by the compiler for best efficiency, but there is no such language yet. So, in the meantime, it is done at initialization time. The object diagram can be fairly elegantly turned into code using the fluent coding style shown in the XR5000 example.

7.11. DSLs

We compared ALA and DSLs in the quick overview here

ALA includes the main idea of DSLs in that the fundamental method "represent[s] requirements by composition of domain abstractions". It shares the DSL property that you can implement a lot more requirements or user stories in a lot less code.

But ALA only tries to be a light-weight way of telling ordinary developers how to organise code written in your underlying language. Although the domain abstractions do form a language and the paradigm interfaces give it a grammar, ALA doesn’t pursue the idea of a language to the point of textural syntactic elegance. Instead, you end up with explicit wiring methods to combine domain entities, or plain old functional composition, or some other form of composition in the wider sense of the word. Often, the text form is only a result of hand translation of an executable diagram. ALA certainly doesn’t overlap with DSLs to the extent of an external DSL, nor does it try to sandbox you from the underlying language. It therefore does not require any parsing and doesn’t need a language workbench, things that may scare away 'plain old C' developers.

Like DSLs, ALA can be highly declarative depending on the paradigm interfaces being used to connect domain abstractions. It is better to have the properties of composition and composability in the your domain language even if they may not be in a perfectly elegant syntactic form. ALA may end up composing abstractions with calls to wireTo methods instead of spaces or dots. But often a diagram using lines is even better than spaces and dots.

In DSLs, it is important that different languages can be combined for different aspects of a problem. For example, a DSL that defines State machines (the state diagram) and a DSL for data organisation (Entity Relationship Diagram) may be needed in the same application. You don’t want to be stuck in one paradigm. ALA recognises this importance by having paradigm interfaces that are more abstract than the domain abstractions.

DSLs probably work by generating a lot of code from templates whereas ALA works by reusing code as instances of abstractions. Both of these methods are fine from the point of view of keeping application specific knowledge in its place, and domain knowledge in its place. Howver, the distinction between ALAs domain layer and programming paradigms layer is probably not so as clearly made in the implementation of the templates.

It is an advantage of DSLs that they can sandbox when needed. An example from the wiring pattern earlier is that the ports of instances do not need to be wired. Therefore, all abstractions need to check if there is something wired to a port before making a call on it. Enforcing this is a problem I have not yet addressed.

A possible solution, albeit inferior to a real DSL that would tell you at design-time, might be that when there are tools that generate wiring code from diagrams, they automatically put stubs on all unwired ports. These stubs either throw an exception at run-time, or just behave inertly.

ALA is different from external DSLs. ALA is just about helping programmers organise their code in a better way. It doesn’t try to make a syntactically elegant language, as a DSL does. Certainly an external DSL will end up representing requirements in a more elegant syntax. But that is not the most important thing in ALA. The most important thing is the separation of code that has knowledge of the requirements, which will cause the invention of abstractions that have zero coupling (because the coupling was really in each requirement - that is why a requirement is cohesive). ALA also avoids taking the average imperative language programmer out of their comfort zone. It does not require a language workbench and does not sandbox you from the underlying language.

ALA probably does fit into the broadest definition of an internal DSL. However, again, it does not target syntactic convenience in the expression of requirements so much as just separating the code that knows about those requirements from the code that implements them. An internal DSL usually aims to have a mini-language that is a subset of the host language, or it tries to extend the host language through clever meta-programming to look as if it has new features. ALA is about abstraction layering. It is about this design-time view of knowledge dependencies: what abstractions in lower layers are needed to understand a given piece of code.

7.12. Dependency injection

7.12.1. Similarities

In ALA you inject run-time required objects via setters.

7.12.2. Differences

ALA uses explicit wiring, never automatic wiring. For one thing, the wiring is required to compose from a pallet of domain abstractions. But secondly, and more importantly, you do not want the knowledge that the wiring represents to disappear into the abstractions themselves, not even as meta-data. That would destroy the abstractions.

In ALA, the explicit wiring can’t be XML or JSON, even if it can be modified at run-time. Usually, because a network structure will be required, the explicit wiring must be a diagram. However, it can be a projection editor, so that the structure is entered in text form (preferably not XML or JSON) and live viewed in graphical form.

In ALA, abstraction pairs don’t have their own interfaces for their instances to communicate. So we don’t have the situation where class A has a dependency on class B, and so an object of class B (or one of its subclasses) is injected into class A. Similarly, we wouldn’t have the situation where class A requires an interface that is implemented by class B.

In ALA the dependencies can only be on paradigm interfaces, which are a whole abstraction layer more abstract. So we need to be thinking that if class A accepts or implements a certain paradigm interface, there could be any number of other abstraction instances that could be attached. Furthermore, we could build arbitrarily large assemblies - composability. Or we don’t have to connect an instance at all. So it doesn’t really make sense to call what we are injecting 'dependencies'. We just think of it as wiring things up, like electronic components.

7.13. Component Based Software Engineering

ALA uses many of the same methods found in component based engineering or the Components and Connector architectural style.

Similarities
  • Components are Abstractions.

  • Reusable software artefacts.

  • Connection ports for I/O.

  • Composability

  • Both instantiate components, specialize them by configuration, and compose them together to make a specific system.

  • ALA’s 3rd layer has interfaces used to wire abstractions in the 2nd layer, so at a lower level (more abstract) level. They represent something more like programming paradigms. The equivalent pattern in components engineering is "Abstract Interactions".

  • The architecture itself is composed of a generic part and a specific part. The general part is the ALA reference architecture itself and the components or the connectors architectural style. The specific part is the wiring diagram of the full system.

Differences
  • Component based engineering technologies such as CORBA primarily solve for platform and language interoperability in distributed system whereas ALA brings some of the resulting concepts and properties to everyday small-scale, non distributed development as well, where the only separation is logical.

  • In ALA there is perhaps more particular emphasis on making components clearly more abstract than the systems they are used in, and making the interfaces clearly more abstract than the components. The components are pushed down a layer and the interfaces down to a layer below that. Then all dependencies must be strictly downwards in these layers. In component based engineering, this structure is not necessarily enforced. If the components are just a decomposition of the system, then the system, components and interfaces may all be at the same level of abstraction, making the system as a whole complex.

  • ALA depends on the 'abstractness property' of components to get logical separation, and so calls them 'Abstractions' and not components to help them retain that property. Even if there will only be one use and one instance, it is still called an abstraction. This keeps them zero coupled and not collaborating with other abstractions they will be wired to.

  • ALA layers are knowledge dependency layers. Components may still be arranged in layers according to run-time dependencies, such as communication stacks. In ALA run-time dependencies are always implemented as explicit wiring inside another higher layer component.

  • ALA’s top layer must be a straight representation of the requirements, whereas components may tend to be decomposed pieces of the system.

  • ALA’s 2nd layer of components are designed for expressiveness of user stories or requirements, and provide DSL-like properties. ALA puts emphasis on the 2nd layer of components having the scope of a domain as the means of explicitly controlling the expressiveness of the pallet of components.

  • ALA is not fractal. In ALA the components of components are abstractions that become more abstract and thus ubiquitous and reusable. ALA therefore uses abstraction layers rather than hierarchies.

  • ALA forces decisions about which abstraction layers the software artefacts go into, and then controls knowledge (semantic) dependencies accordingly.

  • ALA tries to make the abstraction layers discrete and separated by a good margin.

  • ALA puts greater emphasis on wiring being able to represent any programming paradigm that suits the expression of requirements, and the use of many different paradigms in the same wiring diagram.

  • ALA emphasises the cohesion of functional parts of a system such as UI, logic and Data, by bringing them all together in one small diagram using domain level components

  • Instead of 'required' interfaces, in ALA they are called 'accepts' interfaces. This is because the abstractions are more abstract and composable, so, as with Lego blocks, there isn’t necessarily a connection to another instance.

7.13.3. Domain Driven Design

Domain Driven Design’s "Bounded Contexts" and ALA’s Domain Abstractions layer have the same goal, that of encapsulation of the domain specific knowledge.

Domain driven design appears to concentrate on common languages to allow pairs of elements to communicate, which ALA explicitly avoids. ALA tries to abstract the languages so that they are more abstract and fundamental than the domain, and more like programming paradigms.

7.14. Microservices

TBD

7.15. Hexagonal Architecture (Ports and Adapters)

ALA includes the basic idea of hexagonal architecture, but with modification using the Bridge Pattern to keep cohesive knowledge belonging to the application from being split. This was explained in an earlier section of this article. [ComparisonHexagonal]

7.16. Architecture evaluation methods

Methods such as ATAM tell us how to evaluate an architecture for quality attributes such as maintainability, for instance by giving it modification scenarios to test how difficult the modifications would be to implement. There are several scenarios based methods to do this such as ATAM. Using this we could, theoretically, iteratively search over the entire architecture design space to find a satisfactory solution. It’s a bit analogous to numerically solving for the maxima of a complex algebraic formula. In contrast, ALA is analogous to an 'algebraic solution'. If the desired quality attributes, and all the software engineering topics listed above are the equations, ALA is the algebraic solution. It simplifies them down into a parameterised template architecture, ready for you to go ahead and express your requirements.

7.17. Monads

In this article, we talked about monads a few times because they are an important example of composition in software engineering. Also, like ALA, they use the concept of separating (in time) composition from execution. You can bind monads together, and it builds a structure that you can later execute. You can wire instances of domain abstractions together, and it builds a structure that you can later execute. In this respect they are the similar.

When you later execute a monad structure (by calling a function on the last monad you binded with), it (usually) returns its value (or values). When you later execute a wired ALA structure, it (usually) starts running continuously, (usually) waiting for events that can appear at various places in the structure and acting on them. Monads can run continuously as well, as for example in reactive extensions with hot observables. Each monad binding is restricted to a data-flow of a single type, and in a fixed direction. Each ALA wiring is arbitrary in its meaning, according to whatever is most useful to describe requirements. A single wired connection can carry data as needed in both, or the composition may be about something other than data-flow.

Often when monads are used, the execution is done immediately following the binding. So the deferred nature of the execution is not always obvious. I found that the separation between composition and execution of monads to be an important aspect to understand when comparing with ALA composition. In ALA all composition takes place at initialization time. There is a very clear separation between that and run-time.

This much separation is not so common with monads. Monads use the separation primarily as a way to do composition with pure functions, and have all the dirty work contained and abstracted away in well tested reusable classes.

Where you might compose (bind) IObservable or Task monads for almost immediate execution following, in ALA you would tend to compose (wire up) data streams or event sources at initialization time that can then execute many-times thereafter.

Another difference is syntax. Monads are composed using a dot operator, a method call, and configured with lambda function passed to the method:

source.Filter(x=>x>=0).Select(x=>sqrt(x))

This code filters out values from the source that are negative and then calculates the squareroots. In ALA, because composition is generalized, the syntax would look like this:

source.WireIn(new Filter(x=>x>=0)).WireIn(new Select(x=>sqrt(x))

But usually this code is generated from a diagram.

In functional programming, the binding code that builds the structure is pure functions. When you ask the structure to 'execute' all the dirty code is contained inside these reusable abstractions called Monads. The code that constrauts a particular application is clean and free of side effects. ALA makes use of this same property of reuable abstractions, and its wiring code is pure functional.

7.17.1. Understanding monads

Monads are notoriously hard to learn, but they are nice simple insight once you get there. Monads actually seem to have this property that you cannot understand any explanation of them until you first understand them. Thus it is a bootstrapping problem. Here is my experience of going through that bootstrapping process in case it is useful. I am not going to try to explain monads myself, because, even it was possible, others would do that far better than I would.

  1. First understand that Monads are like physics. Physicists explain that you never really understand physics, you just get used it. Unless you are a mathematician or otherwise gifted, the same is true for monads.

  2. The way to get used to new concepts is to read multiple web-sites on the topic. Read each one until you get lost then swap to another one. Keep going like this. For average concepts like design patterns I use this technique and it requires maybe five websites. For monads it took me maybe ten. You will need to return to some of them iteratively to get further each time.

  3. If you don’t know Haskell, prefer the web sites that explain them in the language you already know.

  4. The common essential ideas in those websites will start to embed themselves in your brain.

  5. Eventually, and fairly suddenly, the simple insight that is monads will happen.

I thought few of the web-sites that I used adequately emphasised the monad property of separation (in time) of composition and execution. They did use examples of it such as IEnumerable and Task. They represent what they can do in the future, without actually doing it now. That’s why the binding functions are called bind in the functional world, because it doesn’t (necessarily) do anything except build a structure that can later be executed to actually do the work.

7.18. Reactive Extensions

In ALA, when you wire together

7.19. WPF’s XAML

TBD

7.20. Functional programming

TBD

7.21. Functional programming with monads

TBD

7.22. Functional Reactive Programming

TBD

7.23. Multi-tier Architecture

TBD

7.24. Onion Architecture

TBD

7.25. Clean Architecture

TBD

8. Chapter eight - Surrounding Topics

8.1. Recursive abstractions

ALA enforces a strictly layered (non-circular) knowledge dependency structure. It encourages a small number of abstraction layers at discrete well separated levels of ubiquity as a framework for the knowledge dependencies. This would appear to exclude the possibility of the powerful abstraction composition technique of recursion, where the meaning of an abstraction is defined in terms of itself. (Or an abstraction implementation may need knowledge of another abstraction, which in its turn has an implementation that needs knowledge of the first abstraction. This appears to require circular knowledge dependencies.

Circular knowledge dependencies happen all the time in functional programming where recursion replaces iteration. This is generally when a function needs to call itself or a class needs to use 'new' on itself. For example, a recursive descent compiler will have a function, 'statement', which will implement specific statements such as 'compound statement', 'if statement' and so on, and in those there will be a recursive call to the function, 'statement'. The following Syntax diagram represents part of the implementation of function 'statement'.

diagram 24.png
Figure 28. Syntax Diagram showing implementation of statement using recursion

In ALA, we want to preserve the idea of clear layers defining what knowledge is needed to understand what. Resolving this dilemma could get a bit philosophical. Since abstractions are the first class artefacts of the human brain, it may be best to think about how the brain does it. The brain must actually have two abstractions with the same name but at different levels. The first is analogous to a 'forward declaration' in a language to allow a compiler to know about something that will be referred to in a more abstract way before it finds out about it in a more specific way.

By this analogy, ALA sometimes requires the concept of a forward-declared-abstraction, something that is clearly more abstract than the concrete implementations. Therefore, we can put this forward declaration in the next layer down, just as we would a paradigm interface. In the recursive descent compiler example, we would first have the abstract concept of a statement, meaning a unit of executable code as an interface in a lower layer. Then the specific abstractions, compound statement, if statement and so on are in a higher layer. They both provide and accept the interface.

Another language example is that an expression is composed of terms, a term is composed of factors, and a factor can be composed of expressions (enclosed in round brackets). If we model these compositions as direct knowledge dependencies, we would have too many layers - and they would not be becoming more abstract as we go down. The existence of the recursion at the end reinforces that. It seems that all three, expressions, terms and factors, should have abstract interface versions at a lower level.

Not all cases of recursion would require these interfaces. If, for example, in your old way of doing things there is a long chain of function calls, with the last one calling the first one, all of them are probably run-time dependencies, not knowledge dependencies at all. So in ALA, they should all be changed to be wirable, and wired together by an abstraction in a higher layer. The paradigm interface that is used to allow them to be wired may be,for example, data-flow. So recursion does not necessarily require different interfaces for each different abstraction involved in the circular dependency.

8.2. Abstraction of Port I/O properties

This is an advanced topic that allows abstractions to be written without knowing details of the implementation of the communications. The idea is for the language to support logical or abstracted I/O ports that work for any type of technical communication properties such as described in the sections below. If we allow these properties to be binded late, say at compile-time, they can be changed independently of the domain abstractions. This allows tuning of performance or physical deployment of the abstractions to different processes or hardware.

I have been looking into how this could be accomplished using a conventional language, but it seems quite hard.

8.2.1. Push or Pull

Say an abstraction has a single logical input that can be wired to and a single logical output that can be wired from. Both the input and the output could be used in either a push or a pull manner.

For the input, push means we will be called with the data. Pull means we will call out to get the data.

For the output, push means we will call out with the data, and pull means that something will call us to get the data.

There are four combinations possible:

  • push push : push through

  • pull pull : pull through

  • push pull : internally buffer the input or output

  • pull push : active object

Let’s imagine we have a function that processes the data inside the abstraction.

The four combinations would require the function to run as a result of a function call from the input, or a function call from the output. The function result may be put into an internal buffered or be pushed out. The function may need to receive its input from an input buffer or by pulling. The function may need to run via a 3rd input that is polled or called by a timer.

We could conceivably write an abstract I/O class with an output interface and an input interface and a configuration interface that allows it to be configured late on how to do the I/O. This abstract I/O object would call the function to do the work at the right time according to its configuration.

8.2.2. Synchronous or Asynchronous

8.2.3. Buffered or unbuffered

8.2.4. Shared memory or messaging

8.2.5. Exposed state plus notification

8.2.6. Synchronous Request/Response

8.3. Working with legacy code

In old (non-ALA) legacy code, abstractions, if they ever existed, have usually been destroyed by coupling. If there is no model left by the original designer, or it is out of date, I first create one. I usually have to 'reverse engineer' the model by doing many 'all files searches' and trying to build a mental picture of how everything fits together. It can quickly become mentally taxing if the user story is non-trivial. So I build a UML class diagram from the searches (their one useful application) as the background (using light lines), and a tree of method calls for the specific user story on top of it (using heavier lines). These diagrams can end up looking pretty horrific, because the knowledge of the user-story has become so scattered, especially when inheritance is involved. The tree of method calls will come into the base class but leave from a subclass method.

This process can take several hours to a day for a single user story. Once the code for the single user story is understood, some acceptance tests are put in place for it, by putting in insertion points as close as practical to the inputs and outputs for the user story.

The next step is to factor out the method call tree for the user story into a new abstraction. This typically contains a sequence of new abstract activities or data transformations. These new abstractions are pitched at the domain level. Sometimes, if in C#, I will use Reactive Extensions. The user story may become a single RX sequence. The abstractions are then implemented, with tests, by copying and pasting useful code snippets from the original classes into the new abstractions. The old classes are marked for deprecation.

Conversion of user stories takes place iteratively.

8.4. Writing tests architected in ALA

TBD

8.5. Debugging ALA programs

Because in ALA you can get multiple instances of the same class used in multiple places, and multiple implementations of the same interface used in different places, debugging is easier if the instances are able to identify themselves. For this reason I tend to have a local property in every class called Name. The property is immutable and set by the constructor.

8.6. ALA language features

One of my first hobby programming projects was a compiler for a C-like high level language for embedded systems. At the time I had lots of energy to write the compiler and optimize the object code (written in itself, the performance of both compiling and of object code execution beat the first C compilers to later appear by around a factor of ten) but I lacked a lifetime of experience to design a language. Forty years later, I feel as if it’s partially the other way around, at least for language feature that would support good architecture. The language I should have implemented way back then should have been an ALA language - one that supported ALA architecture by having the needed constraints.

It would have had Abstractions and Instances as first class elements. The name Abstraction is to reinforce the obvious use of the only type of element that the brain uses at design-time for any kind of separation of concerns.

It would support a single type of relationship - a knowledge dependency. You would have to define your four layers, and keep them in separate folders so you would be forced to decide at what abstraction level any given piece of design-time knowledge would go. Of course, it would only allow knowledge dependency relationships from one layer to a lower layer. If you wanted to add an extra layer to the chain of dependencies, that would be a bad design decision. For example, if your application is getting too large, you could create a layer between it and the domain abstractions layer called 'plug-ins'.

Instances would work like components in that they would have ports for I/O. Like interfaces, ports are defined in a lower layer. The only way of instantiating abstractions and connecting them together is inside an abstraction in a higher layer.

Abstractions would support multiple ports of the same interface. Current languages have the difficulty that you can only implement one interface of a given type, which we had to workaround by having connector objects.

Ports would support late configuration of all communication properties such as push, pull, asynchronous, synchronous (explained above) without changing the Abstraction.

Such a language would overcome many of the problems of current languages that encourage non-ALA compliant practices. But the invention of good abstractions in the first sprint of any green-field project would still be a skilled phase requiring an innate ability to abstract.

Feedback

Any feedback about this article is welcomed. Please send to johnspray274<at>gmail<dot>com