The software development landscape is vast. It is so easy to be stunned with how much one needs to learn and people new to this field won’t know what is important to learn first.
Let me help orientate you. While the details of programming languages, libraries, frameworks, databases, cloud infrastructure are all important, we have to understand they are only a means to an end.
Ultimately we are here to solve problems within a given set of constraints, or in other words, we are designers.
Holup, we’re talking about software right? Aren’t designers are those well dressed people who draw things all day? Yes, but consider what “designers” do. They produce designs to solve a problem within a given set of constraints. We just happen to be software designers.
The neat thing about design is, there are universal truths and principals. All design is trying to make everything as simple as possible, but not simpler. Designers from different fields can all learn from each other.
If we strip away the specific requirements and infrastructure details, what are we left? The core algorithm.
Requirements & infrastructure constrain the core algorithm
Software design manifests as the core algorithm. It is constrained by the requirements and infrastructure.
Often for new features, I don’t quite know what the API or database schema needs to look like. I tend to start in the middle and work my way out.
I start with a sketch Domain model and try to write the core algorithm with it. Unit tests are easy to write at this stage as everything tends to be pure functions.
I then see if I can connect the dots from the core algorithm to meet all the requirements on one end. On the other end I see if I can connect the dots to an implementation that works with the infrastructure.
If I discover its not possible to connect all the dots, it reveals information to me on why the domain model or core algorithm is insufficient. Often it’s easy from this point to see how adjustments are to be made to complete the feature.
Let’s look at an ecommerce example.
In this example we’re building an online store and the feature is checkout. We’ll keep the requirements and infrastructure simple to highlight the design process.
Start with the nouns in the requirements and figure out the rough shape.
But what are we missing? There seems to imply there is a concept that binds product & quantity together. And is “Stock” the right noun? That seems to be an outcome more than a noun. Also, since payment/fulfillment are handled afterwards, those systems need something to act upon.
These missing nouns should be discussed with the product owner and requirements should be clarified to include them. After doing so, we learn that the two missing nouns are:
The infrastructure unironically mirrors the stack at Battlefy. At this stage of development of this example, we would ignore the frontend and http layer. We would primarily focus on the service layer, which is detailed in my other post How to write testable code with MongoDB.
The service layer is pure business logic and where the core algorithm resides.
Here is the first stab at the core algorithm. It implements all the requirements but hand-waves away the shipping address/payment details.
We can then write a test to ensure we’ve met the requirements.
Annnnd we’re done and ship it right? Not so fast. While we have achieved functional requirements, there are implicit non-functional requirements.
What happens when the server crashes when an user is in the middle of checkout? Most critically, if the server crashes after updateInventoryStockByProductSku, but before createOrder, then we would have reserved inventory that would never get shipped.
We need checkout to be ACID and fortunately in this made up example, we can use MongoDB transactions.
We can iteratively improve the core algorithm until it meets all the requirements and runs on the infrastructure. This allow ones to explore the solution space and make progress. Sometimes you’ll end up in a dead end, but that’s OK. That is still progress as know now learned of solutions that don’t work.
MongoDB transactions have a bit of a wonky API. We have to start a session and ensure find/update/insert use the session within a transactions. This many seem strange to those who don’t know the intimate details on how databases work, but this is critical for causal consistency and Mongo’s ability to auto-retry transactions.
In order to keep the service layer testable, we will keep the session/transactions specifics in the repository layer. This will complicate the core algorithm with a lot more noise, but I’ve tried to keep the business logic clean.
We need to pass a closure into checkoutTransaction as this closure could be run multiple times if MongoDB detect a transaction commit conflict or failure. This is part of the auto-retry mechanism. The closure is passed a session, which isn’t the raw MongoDB session, but a repository like object with all the find/update/insert commands needed to implement the checkout transaction. For completeness, you can see the repository implementation, but what we really care about is being able to unit test our evolved core algorithm.
The remainder is left as an exercise to the reader as the example as served its purpose. We have shown how a core algorithm developed by relying on the constraints provided by the requirements and infrastructure. The middle is beginning to solidify and all that remains is to the connect the remaining dots to the edges.
Do you want to try your hand at top-down, middle-out and bottom-up design? You’re in luck, Battlefy is hiring.
