26年七月13号，也即是昨天钉钉告警群突然开始疯狂刷屏。

订单同步响应时间一直增加。退款同步开始延迟，业务同事陆续反馈第三方订单迟迟没有同步到下游程序。刚开始我以为只是第三方接口慢了但排查后发现真正的问题出在我们自己的线程池。

原来是我们这边的 thirdparty 微服务里订单同步、退款单同步、订单状态同步这三种业务共用同一个线程池。当天订单状态同步的流量陡增。线程池队列很快被打满，工作线程全部处于忙碌状态。随后提交的退款单同步和订单同步任务开始触发拒绝策略。

共享线程池 = 共享故障域。

一个非主要业务的流量峰值。把主要业务全部拖垮，下游程序大面积数据延迟。

当时我没有任何办法，只能赶紧重新启动。虽然队列里的任务全部丢失。但好在当前写这块代码的程序员把请求先落地到数据库，再去执行程序。

于是我可以从数据库里找到这些记录。并进行比对，重新把数据补充推送。紧急 bug 必须当天处理，我立刻改程序上线，杜绝类似事情 发生。

至于第一层防线，线程池隔离

改造前。三种方法共用一个线程池：

private final ThreadPoolUtils pool = new ThreadPoolUtils;

当订单状态同步流量打满队列后其他业务全部排队或被拒绝。改造后按业务维度拆成三个独立池：

private final ThreadPoolUtils orderSyncPool = new ThreadPoolUtils;private final ThreadPoolUtils refundOrderSyncPool = new ThreadPoolUtils;private final ThreadPoolUtils orderStatusSyncPool = new ThreadPoolUtils;

效果的观点是。 非主要业务的流量再大，也不会影响主要业务；故障被隔离在单个业务线内，不会扩散。按理说，

第二层防线的观点是，CallerRunsPolicy + 结构化告警

原来的自定义拒绝策略只打日志不抛异常。也不执行任务：

// 原拒绝策略：只 log，不执行任务
public void rejectedExecution {
log.warn;不过，// 没有 super.rejectedExecution;// 任务直接丢弃，调用方无感知
}

队列满时任务被静默丢弃；调用方已经返回接收成功，但订单数据根本未处理。按理说，数据丢失，却没人知道——这正是大多数开发者最担心的痛点之一。

I replaced it with a custom CallerRunsPolicy: when queue is full,submitting thread synchronously executes task.

// 自定义 CallerRunsPolicy：先打 ERROR 日志。再同步执行任务
public class CustomCallerRunsPolicy extends ThreadPoolExecutor.CallerRunsPolicy {
private final AtomicInteger callerRunsCount = new AtomicInteger;@Override
public void rejectedExecution {
int count = callerRunsCount.incrementAndGet;StructuredLog.error
.moduleCode
.moduleName
.eventCode
.eventName
.put)
.put.size)
.put
.status
.log;super.rejectedExecution;// 调用父类方法，同步执行任务
}
}

This guarantees that tasks are never lost—only that caller may experience backpressure . In our scenario data integrity trumps speed because third‑party SaaS systems push only once.

再看第三层防线，钉钉独立告警群

I created an independent DingTalk alert group specifically for thread‑pool related alerts.

• The chain is:
• Thread‑pool queue full → CustomCallerRunsPolicy outputs ERROR structured log → Log platform collects → Alert rule matches POOL_FULL event → DingTalk robot pushes to alert group.
• The alert arrives within one minute of event.
• The message contains key info: which pool was full。current active threads,queue size and cumulative trigger count.
• A developer can immediately decide wher to scale up,apply rate limiting or wait for traffic to fall naturally.

第四层防线这方面，数据补偿工具

The classic “server restart” scenario can still wipe out all unprocessed tasks in an in‑memory ArrayBlockingQueue.”

A pre‑processor interceptor writes every request’s parameters into a database table before business logic runs:

// 拦截器预持久化
if {
ThirdDocsRecordDTO record = ThirdDocsRecordDTO.builder
.businessNo)
.servletPath
.param // 完整请求 JSON
.nonce
.build;thirdDocsRecordService.save;return true，}

• This table records every request’s complete parameters.
• A compensation tool queries this table by time range and business number list and replays any unsynced records:

// 数据补偿工具
public void compensate(Date startTime。Date endTime,List docsNoList) {
List records =
recordMapper.selectByTimeRangeAndDocsNos(startTime,endTime,docsNoList);for {
if )) {
continue;// 已处理，跳过
}
replaySync);//
调用同步方法推送
}
}

• The downstream system's deduplication logic ensures idempotency—re‑sending an already synced order will not create duplicates.
• If a server restart wipes out memory queue tasks,this tool can restore m on same day.

小结 & 使用者痛点回顾

• User Pain Point #1: A core business gets throttled by a spike in a non‑core service? Solution: Thread pool isolation per business domain.
• User Pain Point #2: No data loss even when queues are full? Solution: Custom CallerRunsPolicy – tasks run synchronously on submitter thread instead of being discarded.
• User Pain Point #3: Dunno when failure happens?怎么说呢， Solution: Minute‑level DingTalk alerts powered by structured logs.
• User Pain Point #4: If something does get lost。how to recover?怎么说呢， Solution: Persist requests n run a compensation tool to replay unsynced data.
• This four‑layer defense transforms unpredictable crashes into predictable incidents you can monitor and recover from instantly—no more silent failures or blind “data missing” moments.
• Simplified Ops Experience: - Immediate awareness via DingTalk alerts - Same‑day recovery possible with compensation tool - Data integrity guaranteed across all traffic patterns