Message Queues (intro)
Let services talk without waiting on each other, and keep working when one of them falls behind
- explain how a producer, a broker, and a consumer divide the work;
- say concretely what decoupling buys you and what it costs;
- use acknowledgements so a crash never silently drops a message;
- design retries and a dead-letter queue for messages that keep failing;
- reason about at-least-once delivery and why consumers must be idempotent;
- decide when a queue is the right tool — and when it is over-engineering.
1. The problem: a direct call that makes you wait
You already know how to build an Express route that, say, registers a user. Suppose registration must also generate a thumbnail, send a welcome email, and push a record to an analytics service. The naive version does all of that inline, before it answers the client:
app.post('/register', async (req, res) => {
const user = await db.createUser(req.body);
await generateThumbnail(user.avatarUrl); // 800 ms
await sendWelcomeEmail(user.email); // 1200 ms, external SMTP
await analytics.track('signup', user.id); // 400 ms, third-party API
res.json({ ok: true }); // client waited ~2.4 s
});Three things are wrong here. First, the user stares at a spinner for over two seconds even though none of that work needs to finish before we can say “account created.” Second, if the email provider is down, the whole request fails — a transient SMTP hiccup now breaks registration. Third, the request handler is welded to three unrelated systems; a change in any one of them can break the signup path.
A message queue lets you write the durable fact (“user created”) and then hand off the follow-up work as messages. The route returns immediately; the slow, failure-prone work happens elsewhere, on its own schedule, with its own retry policy.