Preparing for the next generation of Spoonity marketing
This year, we launched introductory marketing automation capabilities for a select group of our merchants. The initial feature set focused on user-defined workflows that could respond automatically to end-customer behaviors such as placing an order or leaving feedback.
That launch required a new execution engine and substantial rework across several existing marketing capabilities. It gave us a workable foundation, but it also exposed where the first version of the system would struggle as adoption increased. As we expanded access to more merchants and began building toward multi-channel communication, the constraints became clearer: the limiting factor was no longer downstream delivery providers, but our own execution path.
This post is about that shift. The core problem was not whether we could send messages. It was whether the platform could keep up once campaign volume, workflow complexity, and delivery channels all started growing at the same time.
Revisiting campaigns
Campaigns were one of the first places where those limits became obvious. A one-time bulk send sounds straightforward, but in practice it combines orchestration, audience management, personalization, delivery, and tracking in a single flow. In the first version of the system, those concerns were tightly coupled because speed of delivery mattered more than long-term isolation.
That decision was reasonable at the time. We needed a way to validate workflow-driven campaigns quickly, and an orchestration-first model gave us scheduling, waits, retries, and workflow durability without requiring a custom runtime from day one.
The trouble started once campaign volume increased and the surrounding architecture evolved. Audience materialization, property lookup, stale-data refreshes, per-participant workflow overhead, and dispatch-state writes all accumulated on the hot path. At that point, email providers were no longer the bottleneck. The bottleneck was the amount of work required before a message could even leave the system.
Where we started
The original execution model was built around:
Campaigntable for managing the lifecycle of a campaignCampaignRuntable for tracking progress of discrete "chunks" of a campaignCampaignRunAudiencetable for tracking which subset of the campaign participants were included in a given chunk- Our new workflow engine
- Inline batch processing
- Email delivery through our last-mile partners
At a high level, the legacy flow looked like this:
That architecture had a few defining characteristics:
- One orchestration-driven campaign run
- Legacy runtime centered on
campaignRun/campaignRunAudience - Sequential batch progression represented as a generated workflow
- Pause/resume handled inside the workflow using a
waitForEventparadigm - Personalization performed on the hot path immediately before send
- Email as the primary execution path
The important point is that this model was not failing because it was incorrect. It was failing because it assumed orchestration could continue to absorb more responsibilities as the product expanded.
Scaling Challenges
As campaign volume increased and the runtime became more sophisticated, several constraints emerged. They were related, but not identical, and each one pushed the architecture toward a more explicit execution model.
Audience materialization was necessary, but not sufficient
Persisting the audience before sending was already necessary. First that happened through campaignRunAudience, and later through a dedicated CampaignParticipant table. Materializing the audience gave us deterministic batches and a durable record of who should receive a campaign.
That solved an important correctness problem, but it did not solve the throughput problem. Once execution began, the expensive work was still ahead of us.
Property resolution became a dominant cost
The heaviest part of the pipeline was not provider delivery. It was building the state required to deliver a personalized message.
Each participant could require:
- Loading customer state
- Selecting required properties
- Checking stale data
- Optionally refreshing values from Spoonity or loyalty APIs
- Loading final property values
- Resolving variables into workflow action inputs
In other words, the system could become DB-bound and API-bound long before it became provider-bound. That distinction mattered because it changed where optimization effort would pay off.
Tackling the problems
We did not replace the system all at once. The architecture changed by separating responsibilities that had previously been bundled together, then moving the source of truth for execution away from generated workflow structure and into durable records that could be shared across workers and runtimes.
Multi-channel dispatch foundation
The first pressure on the original design came from channel expansion. Email had been the original execution path, but it was not going to remain the only one. As we began to think about adding push and in-app messaging to the infrastructure, medium-specific logic could no longer live comfortably inside campaign handlers.
That mattered for two reasons. First, each new medium would otherwise duplicate the same execution concepts in a different form. Second, tying orchestration directly to delivery behavior would make scaling uneven and provider integrations harder to isolate.
The architectural decision was to treat messaging as a normalized side effect rather than as the execution model itself. Instead of letting each campaign path decide how delivery should work, the platform needed a shared dispatch layer that all workflow actions could target.
We wanted to ensure the architecture could handle any mediums before we finished adding them, and so took steps before they were ready to prepare.
That led to three core tables:
MessageDispatchfor tracking individual messages linked to a channelMessageDispatchEventfor tracking message events such as deliveries and opens for each messageMessageChannelfor normalizing supported messaging mediums
We also added the first application-side handlers needed for push and in-app messaging. The result was not just broader channel support. It was a cleaner boundary: workflow execution could produce communication work, and channel-specific systems could own the details of delivering it.
Durable execution entities
Once messaging had a cleaner boundary, the next problem became harder to ignore: the runtime itself was still shaped around legacy campaign artifacts. campaignRun and campaignRunAudience were useful for bootstrapping bulk sends, but they did not express execution as clearly as the platform now required.
That distinction mattered because reliability features such as retries, recovery, cancellation, and cross-workflow reuse all depend on where the system stores truth. If progress lives mainly in orchestration structure, those operations stay awkward. If progress lives in durable execution records, the rest of the system becomes easier to reason about.
The decision was to introduce execution-native entities:
CampaignExecutionCampaignParticipantActionExecution
Together, these gave the platform:
- A durable campaign execution root
- Participant-level status
- Shared execution records across campaigns and journeys
- Communication side-effect tracking separated from workflow action execution
This was the most important conceptual shift in the system. CampaignExecution became the root record for a send, CampaignParticipant became the durable record of recipient progress, and ActionExecution became the common operational layer for workflow actions. At that point, the source of truth started moving away from batch registration artifacts and toward explicit execution state.
Separating audience generation from execution
Early versions of the execution pipeline concentrated too much responsibility in a single orchestration flow. A campaign would bootstrap its execution, generate participants, resolve execution context, and begin dispatching messages as part of the same coordinated process.
While this approach worked well for smaller campaigns, it became increasingly difficult to scale. Audience generation, participant execution, and message delivery are fundamentally different workloads. They consume different resources, have different throughput characteristics, and benefit from different scaling strategies.
Rather than continuing to optimize a single execution path, we separated the pipeline into independent stages connected through durable state.
The resulting execution flow looks like this:
This wasn't simply a refactoring. It fundamentally changed how work moved through the system.
Instead of a single orchestration path performing every operation inline, each stage now completes its responsibility before handing execution to the next. Those handoffs are backed by durable execution records, allowing each stage to scale independently while preserving progress, retries, and observability.
The separation also made it possible to optimize each stage independently. Audience generation focuses on efficiently materializing participants, execution focuses on workflow progression, and delivery workers focus exclusively on communicating with external providers.
Worker-based dispatch
Once execution was staged, dispatch no longer belonged inside orchestration. Keeping delivery inline would have pulled medium-specific side effects back into the same path we were trying to simplify.
That mattered operationally as much as architecturally. Different channels have different delivery providers, different throughput profiles, and different failure modes. Treating them as independent worker concerns makes the system easier to scale and easier to debug.
The decision was to move dispatch into dedicated workers and let orchestration focus on coordination.
The main worker families now include:
audience-generation-workerparticipant-execution-workeremail-workerpush-workerin-app-worker
With those roles separated, the execution pipeline became easier to read:
That structure gave us queue-based fan-out, medium-specific execution isolation, worker-specific scaling and deployment, and tighter control over provider integrations. More importantly, it kept the execution model coherent as new delivery paths were added.
Database-driven cancellation
Earlier behavior depended too heavily on workflow-engine-owned state, which introduced latency into state checks and made cancellation harder to enforce uniformly across all execution paths.
As work became distributed across more stages and workers, cancellation had to become more explicit. A workflow-engine-centric model could stop some work, but it was not a reliable control plane for stopping all future work once execution was spread across orchestration, participant processing, and delivery workers.
The practical issue was consistency. If different parts of the runtime checked different forms of state, cancellation would always lag somewhere.
The decision was to make persistent database state the authority for cancellation. Instead of relying primarily on workflow run references, the system now uses states such as:
JourneyRun.statusCampaignExecution.statusCampaignParticipant.status
That makes cancellation enforceable across:
- Orchestrator actions
- Participant execution
- Dispatch workers
- Reconciliation paths
The result is not just better stop behavior. It is a clearer operational model: future work is gated by first-party execution state, not by assumptions about where the workflow runtime is in its own lifecycle.
Finding the right execution model
By this point, the platform had become significantly more durable and observable. Campaign execution was now backed by persistent state, responsibilities were clearly separated across workers, and failures could be recovered without losing progress.
The next challenge was throughput.
The question was no longer whether the platform could execute campaigns reliably—it could. The question was how much work each workflow execution should own. Put another way, what was the right execution granularity?
Rather than assuming the answer, we experimented with multiple execution models, each representing a different balance between scalability, traceability, and operational overhead.
Experiment 1: One workflow per participant
Our first approach treated every participant as an independent workflow execution. Each invocation of campaign.participant.execute owned the complete lifecycle of a single recipient.
Architecturally, this model was compelling. Every participant had an isolated execution history, retries were naturally scoped to a single recipient, and failures could be recovered independently without affecting the rest of the campaign. The execution model was also easy to reason about because each workflow represented exactly one unit of work.
However, isolating every participant also exposed the true cost of orchestration.
Processing a single participant involved considerably more work than simply sending a message. A typical execution included:
- Participant state reads
- Execution-state validation
- Action-execution creation
- Message-dispatch creation
- Message-dispatch event persistence
- Worker enqueueing
- Reconciliation and completion updates
None of these operations were unnecessary. They existed to provide the durability, observability, and recoverability that had become core design goals of the platform.
The challenge was scale.
When multiplied across tens or hundreds of thousands of participants, the orchestration overhead itself became a meaningful portion of the total execution time. Although each workflow remained lightweight, coordinating one workflow per recipient generated a significant amount of database activity, scheduling work, and workflow bookkeeping.
The model delivered excellent correctness, but its operational cost grew almost perfectly linearly with audience size.
Experiment 2: One workflow per batch
To reduce that overhead, we shifted the execution boundary.
Instead of assigning a workflow to every participant, a single invocation of campaign.participant-batch.process became responsible for processing a batch of participants. Within that workflow, specialized batchActionHandlers executed participant actions while sharing the same orchestration context.
This seemingly small architectural change had a significant impact.
Instead of paying the workflow coordination cost for every recipient, the platform amortized that cost across an entire batch. Database reads, execution coordination, and workflow scheduling became dramatically more efficient while preserving participant-level execution state and message tracking.
Importantly, batching did not sacrifice observability. Participants continued to maintain independent execution records, message dispatches, and reconciliation state. The optimization was entirely in how work was orchestrated, not in the execution guarantees themselves.
The result was an execution model that preserved the reliability and traceability of participant-level processing while scaling far more efficiently. Batch execution ultimately became the foundation of the platform because it struck the right balance between operational efficiency, execution correctness, and long-term scalability.
Where we are now
After multiple iterations, the architecture naturally converged into a hybrid execution model.
Rather than placing every responsibility inside the workflow engine or the database, each component now owns the responsibilities it performs best. Workflows coordinate execution, the database provides durable state, and specialized workers handle delivery.
The result is an execution pipeline that scales independently while remaining observable, recoverable, and resilient to failures.
The most important architectural boundary is the separation between orchestration and durable execution.
The workflow engine remains responsible for coordinating execution:
- campaign orchestration
- waits and scheduling
- participant or batch progression
- action sequencing
- orchestration control
The database owns the execution model itself:
- durable participant state
- action execution records
- message dispatch state
- worker queueing
- database-driven cancellation
- reconciliation
This distinction keeps workflows lightweight while allowing execution state to survive retries, deployments, and failures.
Key takeaways
Messaging is not the execution model
One of the biggest architectural shifts was recognizing that message delivery is only one type of workflow action.
Treating messaging as a side effect instead of the execution model allowed us to build a durable execution foundation that can support additional action types as the platform evolves.
ActionExecution became the shared operational center, while MessageDispatch became a communication-specific concern.
Durable participant state changes everything
Replacing campaignRunAudience with CampaignParticipant fundamentally changed how execution is modeled.
Participants became long-lived execution entities rather than temporary audience artifacts, enabling reliable retries, progress tracking, reconciliation, and cancellation throughout the execution lifecycle.
Worker isolation enables independent scaling
Separating delivery workers from orchestration reduced coupling across the platform.
Each messaging channel can now scale, deploy, and recover independently without increasing orchestration complexity, while provider-specific integrations remain isolated behind dedicated workers.
Keep expensive work out of the hot path
One recurring lesson throughout this project was that broad property resolution does not belong inside high-throughput execution paths.
Moving toward required-property discovery and participant context snapshots keeps campaign execution predictable while dramatically reducing unnecessary database work during message processing.
Database state is the control plane
Making the database the source of truth for execution state transformed cancellation into a deterministic operation.
Rather than relying on transient workflow state, future work is controlled through persistent execution records. This makes cancellation reliable across retries, worker restarts, and deployments, turning it into a foundational reliability feature rather than an implementation detail.
Where do we go from here?
The current architecture is significantly stronger than the original one, but our work doesn't stop here.
So far, we've managed to improve performance by 3-5x and have a clear plan to continue to improve productivity and response times to meet the demands of enterprise merchants.
Additionally, going beyond performance, there are several feature areas that we are excited about that we believe will unlock tremendous value for our merchants. In our Next Generation Platform, we plan to add even more messaging channels, as well as introduce multi-step messaging to allow for the creation of campaigns that can react to end-customers' behavior on the fly.
We've already begun work on some of these, and are excited to have more formal feature announcements about the Next Generation Platform soon. We also plan on releasing more of these deep-dives into the Spoonity architecture over time.