Architecture Patterns with Python

we present four key design patterns: The Repository pattern, an abstraction over the idea of persistent storage The Service Layer pattern to clearly define where our use cases begin and end The Unit of Work pattern to provide atomic operations The Aggregate pattern to enforce the integrity of our data

Whenever we have a business concept that has data but no identity, we often choose to represent it using the Value Object pattern. A value object is any domain object that is uniquely identified by the data it holds; we usually make them immutable: OrderLine is a value object @dataclass( frozen = True) class OrderLine: orderid: OrderReference sku: ProductReference qty: Quantity One of the nice things that dataclasses (or namedtuples) give us is value equality, which is the fancy way of saying, “Two lines with the same orderid, sku, and qty are equal.”

Python’s __eq__ magic method defines the behavior of the class for the = = operator.

being able to use sorted() on our list of batches is nice, idiomatic Python. To make it work, we implement __gt__ on our domain model:

Domain modeling This is the part of your code that is closest to the business, the most likely to change, and the place where you deliver the most value to the business. Make it easy to understand and modify. Distinguish entities from value objects A value object is defined by its attributes. It’s usually best implemented as an immutable type. If you change an attribute on a Value Object, it represents a different object. In contrast, an entity has attributes that may vary over time and it will still be the same entity. It’s important to define what does uniquely identify an entity (usually some sort of name or reference field). Not everything has to be an object Python is a multiparadigm language, so let the “verbs” in your code be functions. For every FooManager, BarBuilder, or BazFactory, there’s often a more expressive and readable manage_foo(), build_bar(), or get_baz() waiting to happen. This is the time to apply your best OO design principles Revisit the SOLID principles and all the other good heuristics like “has a versus is-a,” “prefer composition over inheritance,” and so on. You’ll also want to think about consistency boundaries and aggregates

Domain services are not the same thing as the services from the service layer, although they are often closely related. A domain service represents a business concept or process, whereas a service-layer service represents a use case for your application. Often the service layer will call a domain service.

we want our domain model to have no dependencies whatsoever. 1 We don’t want infrastructure concerns bleeding over into our domain model and slowing our unit tests or our ability to make changes. Instead, as discussed in the introduction, we’ll think of our model as being on the “inside,” and dependencies flowing inward to it; this is what people sometimes call onion architecture

Ports and adapters came out of the OO world, and the definition we hold onto is that the port is the interface between our application and whatever it is we wish to abstract away, and the adapter is the implementation behind that interface or abstraction. Now Python doesn’t have interfaces per se, so although it’s usually easy to identify an adapter, defining the port can be harder. If you’re using an abstract base class, that’s the port. If not, the port is just the duck type that your adapters conform to and that your core application expects—the function and method names in use, and their argument names and types.

If your app is just a simple CRUD (create-read-update-delete) wrapper around a database, then you don’t need a domain model or a repository.

Repository Pattern Recap Apply dependency inversion to your ORM Our domain model should be free of infrastructure concerns, so your ORM should import your model, and not the other way around. The Repository pattern is a simple abstraction around permanent storage The repository gives you the illusion of a collection of in-memory objects. It makes it easy to create a FakeRepository for testing and to swap fundamental details of your infrastructure without disrupting your core application.

Tests that use mocks tend to be more coupled to the implementation details of the codebase. That’s because mock tests verify the interactions between things: did we call shutil.copy with the right arguments? This coupling between code and test tends to make tests more brittle, in our experience. Overuse of mocks leads to complicated test suites that fail to explain the code.

We’ve successfully split our tests into two broad categories: tests about web stuff, which we implement end to end; and tests about orchestration stuff, which we can test against the service layer in memory.

Once you implement domain modeling and the service layer, you really actually can get to a stage where unit tests outnumber integration and end-to-end tests by an order of magnitude.

Rules of Thumb for Different Types of Test Aim for one end-to-end test per feature This might be written against an HTTP API, for example. The objective is to demonstrate that the feature works, and that all the moving parts are glued together correctly. Write the bulk of your tests against the service layer These edge-to-edge tests offer a good trade-off between coverage, runtime, and efficiency. Each test tends to cover one code path of a feature and use fakes for I/ O. This is the place to exhaustively cover all the edge cases and the ins and outs of your business logic. 1 Maintain a small core of tests written against your domain model These tests have highly focused coverage and are more brittle, but they have the highest feedback. Don’t be afraid to delete these tests if the functionality is later covered by tests at the service layer. Error handling counts as a feature Ideally, your application will be structured such that all errors that bubble up to your entrypoints (e.g., Flask) are handled in the same way. This means you need to test only the happy path for each feature, and to reserve one end-to-end test for all unhappy paths (and many unhappy path unit tests, of course).

“Don’t mock what you don’t own” is a rule of thumb that forces us to build these simple abstractions over messy subsystems. This has the same performance benefit as mocking the SQLAlchemy session but encourages us to think carefully about our designs.

the context manager makes it easy to see, visually, what blocks of code are grouped together atomically.

Unit of Work Pattern Recap The Unit of Work pattern is an abstraction around data integrity It helps to enforce the consistency of our domain model, and improves performance, by letting us perform a single flush operation at the end of an operation. It works closely with the Repository and Service Layer patterns The Unit of Work pattern completes our abstractions over data access by representing atomic updates. Each of our service-layer use cases runs in a single unit of work that succeeds or fails as a block. This is a lovely case for a context manager Context managers are an idiomatic way of defining scope in Python. We can use a context manager to automatically roll back our work at the end of a request, which means the system is safe by default. SQLAlchemy already implements this pattern We introduce an even simpler abstraction over the SQLAlchemy Session object in order to “narrow” the interface between the ORM and our code. This helps to keep us loosely coupled.

a constraint is a rule that restricts the possible states our model can get into, while an invariant is defined a little more precisely as a condition that is always true.

Each layer of indirection has a price in terms of complexity and duplication in our code and will be confusing to programmers who’ve never seen these patterns before. If your app is essentially a simple CRUD wrapper around a database and isn’t likely to be anything more than that in the foreseeable future, you don’t need these patterns. Go ahead and use Django, and save yourself a lot of bother.

Rule of thumb: if you can’t describe what your function does without using words like “then” or “and,” you might be violating the SRP.

An event is a kind of value object. Events don’t have any behavior, because they’re pure data structures. We always name events in the language of the domain, and we think of them as part of our domain model. We could store them in model.py, but we may as well keep them in their own file (this might be a good time to consider refactoring out a directory called domain so that we have domain/ model.py and domain/ events.py):

In general, if you’re implementing domain events, don’t raise exceptions to describe the same domain concept. As you’ll see later when we handle events in the Unit of Work pattern, it’s confusing to have to reason about events and exceptions together.

The magic words “When X, then Y” often tell us about an event that we can make concrete in our system. Treating events as first-class things in our model helps us make our code more testable and observable, and it helps isolate concerns.

Events can help with the single responsibility principle Code gets tangled up when we mix multiple concerns in one place. Events can help us to keep things tidy by separating primary use cases from secondary ones. We also use events for communicating between aggregates so that we don’t need to run long-running transactions that lock against multiple tables. A message bus routes messages to handlers You can think of a message bus as a dict that maps from events to their consumers. It doesn’t “know” anything about the meaning of events; it’s just a piece of dumb infrastructure for getting messages around the system. Option 1: Service layer raises events and passes them to message bus The simplest way to start using events in your system is to raise them from handlers by calling bus.handle( some_new_event) after you commit your unit of work. Option 2: Domain model raises events, service layer passes them to message bus The logic about when to raise an event really should live with the model, so we can improve our system’s design and testability by raising events from the domain model. It’s easy for our handlers to collect events off the model objects after commit and pass them to the bus. Option 3: UoW collects events from aggregates and passes them to message bus Adding bus.handle( aggregate.events) to every handler is annoying, so we can tidy up by making our unit of work responsible for raising events that were raised by loaded objects. This is the most complex design and might rely on ORM magic, but it’s clean and easy to use once it’s set up.

the Preparatory Refactoring workflow, aka “Make the change easy; then make the easy change”:

Like events, commands are a type of message—instructions sent by one part of a system to another. We usually represent commands with dumb data structures and can handle them in much the same way as events. The differences between commands and events, though, are important. Commands are sent by one actor to another specific actor with the expectation that a particular thing will happen as a result. When we post a form to an API handler, we are sending a command. We name commands with imperative mood verb phrases like “allocate stock” or “delay shipment.” Commands capture intent. They express our wish for the system to do something. As a result, when they fail, the sender needs to receive error information. Events are broadcast by an actor to all interested listeners. When we publish BatchQuantityChanged, we don’t know who’s going to pick it up. We name events with past-tense verb phrases like “order allocated to stock” or “shipment delayed.” We often use events to spread the knowledge about successful commands. Events capture facts about things that happened in the past. Since we don’t know who’s handling an event, senders should not care whether the receivers succeeded or failed.

from tenacity import Retrying, RetryError, stop_after_attempt, wait_exponential

we should think in terms of verbs, not nouns. Our domain model is about modeling a business process. It’s not a static data model about a thing; it’s a model of a verb. So instead of thinking about a system for orders and a system for batches, we think about a system for ordering and a system for allocating, and so on. When we separate things this way, it’s a little easier to see which system should be responsible for what. When thinking about ordering, really we want to make sure that when we place an order, the order is placed. Everything else can happen later, so long as it happens.

Like aggregates, microservices should be consistency boundaries. Between two services, we can accept eventual consistency, and that means we don’t need to rely on synchronous calls. Each service accepts commands from the outside world and raises events to record the result. Other services can listen to those events to trigger the next steps in the workflow.

This insight is key to understanding why reads can be safely inconsistent: we’ll always need to check the current state of our system when we come to allocate, because all distributed systems are inconsistent. As soon as you have a web server and two customers, you have the potential for stale data.

The so-called SELECT N + 1 problem is a common performance problem with ORMs: when retrieving a list of objects, your ORM will often perform an initial query to, say, get all the IDs of the objects it needs, and then issue individual queries for each object to retrieve their attributes. This is especially likely if there are any foreign-key relationships on your objects.

declaring an explicit dependency is an example of the dependency inversion principle—rather than having an (implicit) dependency on a specific detail, we have an (explicit) dependency on an abstraction:

We’ve used MailHog as a real-ish email server for our Docker dev environment:

When building the availability service, we used a technique called event interception to move functionality from one place to another. This is a three-step process: Raise events to represent the changes happening in a system you want to replace. Build a second system that consumes those events and uses them to build its own domain model. Replace the older system with the new.

When deploying an event-driven system, start with a “walking skeleton.” Deploying a system that just logs its input forces us to tackle all the infrastructural questions and start working in production.

Our first production deployment was a tiny system that could receive a batch_created event and log its JSON representation. This is the “Hello World” of event-driven architecture. It forced us to deploy a message bus, hook up a producer and consumer, build a deployment pipeline, and write a simple message handler.

Figuring out how to model your domain is a complex task that’s the subject of many decent books in its own right. We like to use interactive techniques like event storming and CRC modeling, because humans are good at collaborating through play. Event modeling is another technique that brings engineers and product owners together to understand a system in terms of commands, queries, and events. Tip Check out www.eventmodeling.org and www.eventstorming.org for some great guides to visual modeling of systems with events. The goal is to be able to talk about the system by using the same ubiquitous language, so that you can agree on where the complexity lies.

In practice you will want to make handlers idempotent so that calling them repeatedly with the same message will not make repeated changes to state. This is a key technique for building reliability, because it enables us to safely retry events when they fail.

The components of our architecture and what they all do: - Entity: A domain object whose attributes may change but that has a recognizable identity over time. - Value object: An immutable domain object whose attributes entirely define it. It is fungible with other identical objects. - Aggregate: Cluster of associated objects that we treat as a unit for the purpose of data changes. Defines and enforces a consistency boundary. - Event: Represents something that happened. - Command: Represents a job the system should perform. - Service Layer: Defines the jobs the system should perform and orchestrates different components. - Handler: Receives a command or an event and performs what needs to happen. - Unit of work: Abstraction around data integrity. Each unit of work represents an atomic update. Makes repositories available. Tracks new events on retrieved aggregates. - Message bus (internal): Handles commands and events by routing them to the appropriate handler. - Adapters (Secondary): Concrete implementations of an interface that goes from our system to the outside world (I/O). - Repository: Abstraction around persistent storage. Each aggregate has its own repository. - Event publisher: Pushes events onto the external message bus. - Entrypoints (Primary adapters): Translate external inputs into calls into the service layer. - Web: Receives web requests and translates them into commands, passing them to the internal message bus. - Event consumer: Reads events from the external message bus and translates them into commands, passing them to the internal message bus. - N/A External message bus (message broker): A piece of infrastructure that different services use to intercommunicate, via events.

The basic folder structure looks like this: Project tree . ├──Dockerfile  ├──Makefile  ├──README.md ├──docker-compose.yml  ├──license.txt ├──mypy.ini ├──requirements.txt ├──src  │ ├──allocation │ │ ├──__init__. py │ │ ├──adapters │ │ │ ├──__init__. py │ │ │ ├──orm.py │ │ │ └──repository.py │ │ ├──config.py │ │ ├──domain │ │ │ ├──__init__. py │ │ │ └──model.py │ │ ├──entrypoints │ │ │ ├──__init__. py │ │ │ └──flask_app.py │ │ └──service_layer │ │ ├──__init__. py │ │ └──services.py │ └──setup.py  └──tests  ├──conftest.py  ├──e2e │ └──test_api.py ├──integration │ ├──test_orm.py │ └──test_repository.py ├──pytest.ini  └──unit ├──test_allocate.py ├──test_batches.py └──test_services.py

Pro tip: if you’re mounting volumes to share source folders between your local dev machine and the container, the PYTHONDONTWRITEBYTECODE environment variable tells Python to not write .pyc files, and that will save you from having millions of root-owned files sprinkled all over your local filesystem, being all annoying to delete and causing weird Python compiler errors besides.

Things like the Django admin, which are so awesome when you start out, become actively dangerous if the whole point of your app is to build a complex set of rules and modeling around the workflow of state changes. The Django admin bypasses all of that.

So what should you do if you want to apply some of the patterns in this book to a Django app? We’d say the following: The Repository and Unit of Work patterns are going to be quite a lot of work. The main thing they will buy you in the short term is faster unit tests, so evaluate whether that benefit feels worth it in your case. In the longer term, they decouple your app from Django and the database, so if you anticipate wanting to migrate away from either of those, Repository and UoW are a good idea. The Service Layer pattern might be of interest if you’re seeing a lot of duplication in your views.py. It can be a good way of thinking about your use cases separately from your web endpoints. You can still theoretically do DDD and domain modeling with Django models, tightly coupled as they are to the database; you may be slowed by migrations, but it shouldn’t be fatal. So as long as your app is not too complex and your tests not too slow, you may be able to get something out of the fat models approach: push as much logic down to your models as possible, and apply patterns like Entity, Value Object, and Aggregate. However, see the following caveat. With that said, word in the Django community is that people find that the fat models approach runs into scalability problems of its own,

Suppose you’re working on a Django project that you’re not sure is going to get complex enough to warrant the patterns we recommend, but you still want to put a few steps in place to make your life easier, both in the medium term and if you want to migrate to some of our patterns later. Consider the following: One piece of advice we’ve heard is to put a logic.py into every Django app from day one. This gives you a place to put business logic, and to keep your forms, views, and models free of business logic. It can become a stepping-stone for moving to a fully decoupled domain model and/ or service layer later. A business-logic layer might start out working with Django model objects and only later become fully decoupled from the framework and work on plain Python data structures. For the read side, you can get some of the benefits of CQRS by putting reads into one place, avoiding ORM calls sprinkled all over the place. When separating out modules for reads and modules for domain logic, it may be worth decoupling yourself from the Django apps hierarchy. Business concerns will cut across them.

Validate as little as possible. Read only the fields you need, and don’t overspecify their contents. This will help your system stay robust when other systems change over time. Resist the temptation to share message definitions between systems: instead, make it easy to define the data you depend on. For more info, see Martin Fowler’s article on the Tolerant Reader pattern.

Validation Recap Validation means different things to different people When talking about validation, make sure you’re clear about what you’re validating. We find it useful to think about syntax, semantics, and pragmatics: the structure of messages, the meaningfulness of messages, and the business logic governing our response to messages. Validate at the edge when possible Validating required fields and the permissible ranges of numbers is boring, and we want to keep it out of our nice clean codebase. Handlers should always receive only valid messages. Only validate what you require Use the Tolerant Reader pattern: read only the fields your application needs and don’t overspecify their internal structure. Treating fields as opaque strings buys you a lot of flexibility. Spend time writing helpers for validation Having a nice declarative way to validate incoming messages and apply preconditions to your handlers will make your codebase much cleaner. It’s worth investing time to make boring code easy to maintain. Locate each of the three types of validation in the right place Validating syntax can happen on message classes, validating semantics can happen in the service layer or on the message bus, and validating pragmatics belongs in the domain model.