Exchange – how are Data Exchanged and Stored?

Our blog series has described the Interact and Communicate checkpoints in recent posts. Fitting this topic area is the Exchange checkpoint, in which we will take a closer look at the topics of data exchange formats, persistent data storage, and caching.

What Can I Expect from this Stop?

The Exchange checkpoint includes the following topics:

Data Formats & Read/Write Models
Data Persistence
Data Caching
Multi-tenancy

The full focus of this checkpoint is on how data is exchanged between (sub-)systems and how and where data is persisted or cached.

In distributed architectures with loosely coupled systems, data persistence should also be loosely coupled. Therefore, we generally recommend adopting the “Database per Service” pattern. This pattern is very well suited for greenfield projects; however, in companies with established data assets and data management, an uncoordinated transition to this pattern would pose a significant operational risk. Therefore, at this checkpoint, we support not only the introduction of cloud-native patterns and storage options but also the transition to them.

Data formats form the basis for interoperability between systems. Our approach to this topic is to rely on existing standards wherever possible and to extend them where appropriate.

Using the example of event-based communication between systems, the following questions illustrate the working method of this checkpoint:

Definition of the event format per communication channel:
Event-Notification vs Event-Carried-State-Transfer vs Domain-Event
Possible introduction of separate read models for existing write models:
Consideration of patterns for distributed transactions (e.g., Event Sourcing vs Outbox Pattern)
Storage in distributed caches vs. proprietary persistent data storage

Why should I Stop at this Stop?

As mentioned at the outset, it is essential for the interoperability of systems that systems “understand” each other, and this is made possible by ensuring that formats can be serialized and deserialized by all participating systems.

Well-considered data formats also contribute to sustainable data exchange and data storage, thus forming the basis for long-lived systems (meant in a positive sense 😉). Conversely, unstable formats or exotic persistence technologies can have massive impacts on maintainability, stability, observability, and performance.

Furthermore, this checkpoint helps to fully exploit the possibilities of dynamic scaling. For example, it is almost impossible to dynamically scale a system if system integration is performed at the database level (mutual locking, unclear data sovereignty, etc.).

How Does this Stop Work?

As indicated in the previous paragraph, system integration at the database level is extremely problematic. Therefore, one aspect of this checkpoint, if such a constellation exists, is to overcome system integration at the database level and transition it to communication paths that enable a cloud-native architecture.

System context diagrams provide information about which data formats should or must be supported in communication between (sub-)systems and thus offer a good starting point for the work during this checkpoint. They also help to develop appropriate strategies for any necessary multi-tenancy of the systems.

Considering the CAP theorem, in combination with the architectural goals, dictates the further direction for establishing formats, persistent storage, and (distributed) caches.

Further refinement regarding clear boundaries of responsibility and data sovereignty can be derived from the Context Map. Based on decisions about how the system interacts, data formats ideally tailored to the specific target architecture can be designed.

The attentive reader will probably have noticed at this point that all results from the previously described checkpoints (Architect, Interact, Communicate) flow into this checkpoint and have significant effects on it. However, since these checkpoints are not “silos,” they are carried out with focus, but always with the flexibility to incorporate topics from other checkpoints when sensible and necessary.

This closes the circle of the more software architecture-heavy checkpoints. The next checkpoints aim towards Delivery & Experience.

In fact, we often divide or offer the entire Cloud Native App Journey in two streams. One would be the “Architecture & Design” stream, including Architect, Interact, Communicate, and Exchange, and the other the “Delivery & Experience” stream, including Protect, Deliver, Observe, and Experience.

Thus, we have reached the end of one checkpoint and are ready to continue the journey to the next. I gladly welcome your suggestions for the next checkpoint at this point.

Simply send an email to konrad.renner@fullstacks.eu 😉