MiFaZhan
2bec257289
iotlcm-worker-svr：异步执行中枢（RabbitMQ 消费者 + 全量同步编排）。 - Go module：iotlcm-worker-svr → git.5iot.com/iotlcm-public/iotlcm-worker-svr（214 文件） - 凭据与内部标识脱敏：MySQL/Redis 凭据、私网与生产 IP、内网域名、 库名 xiujie_admin → iotlcm、哨兵名占位 - 159 处真实手机号 → 合法 11 位编造号（保留号段与后 4 位，维持互异， 不破坏「校验 MSISDN 格式」等测试断言） - 移除内部资产：CI 触发器、Jenkins 触发测试文档、预演门禁与镜像 digest 记录 - Dockerfile 改用公共基础镜像（golang:1.25-alpine → alpine:3.23）， 移除写死的 git 凭据 - README 重写（架构/全量同步模型/配置/已知失败），补 Apache-2.0 LICENSE，LF 行尾 + .gitattributes 已知：3 个测试包在源仓同一提交上同样失败（client/upstream、 service/allocate、service/inbound），属既有缺陷，本仓未引入亦未修复。
125 lines
6.6 KiB
Markdown
125 lines
6.6 KiB
Markdown
# Full-sync shared contracts
|
|
|
|
This package establishes the boundary for later queue accounting, archive,
|
|
replay, and HTTP work. It does not change running service behavior. Production
|
|
code in this package does not import the queue package; compatibility tests
|
|
compare its models and keys with the existing queue implementation.
|
|
|
|
## Compatibility and counters
|
|
|
|
- `RunMeta` retains all existing JSON fields and adds optional fields. Existing
|
|
`zset.RunMeta`, `zset.Task`, and `zset.FailureRecord` remain unchanged; adapters
|
|
are a later integration step. The new `RunStatus` is a named string type.
|
|
- `Validate` accepts missing extension fields on old records.
|
|
`ValidateAccounting` is opt-in for runs with known extended counters and
|
|
enforces `enqueuedCount = successCount + failedCount + activeCount + nextRoundCount`.
|
|
`inflightCount` is a subset of `activeCount`.
|
|
- Source cleanup changes `pendingFailureCount`, never historical `failedCount`.
|
|
- Round statistics have no task/card/channel dimensions. `ErrorCounts`
|
|
normalizes both JSON input and output to eight canonical categories, merging
|
|
unknown names into `other` and rejecting negative/overflowing totals.
|
|
- Existing failure payloads retain their free-form classification and message.
|
|
The HTTP `Failure` projection normalizes categories and bounds messages to
|
|
2,048 UTF-8 bytes. Historical blocked tasks may have round zero.
|
|
|
|
## Status domains
|
|
|
|
| Domain | Wire values |
|
|
| --- | --- |
|
|
| Run | `producing`, `running`, `completed`, `completed_with_failures` |
|
|
| Sync type | `normal`, `replay` |
|
|
| Archive | `disabled`, `pending`, `processing`, `succeeded`, `failed` |
|
|
| Source replay disposition | `none`, `pending`, `processing`, `completed`, `needs_attention` |
|
|
| Replay action | `accepted`, `processing`, `cleanup_pending`, `receipt_pending`, `completed`, `needs_attention` |
|
|
| Error category | `timeout`, `rate_limited`, `authentication`, `validation`, `upstream`, `transport`, `internal`, `other` |
|
|
|
|
The source field is `replayDisposition`, distinct from the replay action's
|
|
`status`. Lease tokens, record versions, action revision, idempotency key, and
|
|
request fingerprint are internal and excluded from HTTP JSON.
|
|
|
|
## Interfaces and persistence obligations
|
|
|
|
HTTP consumers depend on `RunReader`, `FailurePager`, `ReplayRequester`, and
|
|
`ActionReader`. Separate live/archive readers support fallback only after a live
|
|
`ErrNotFound`; storage failures must remain visible. An archived summary cannot
|
|
provide per-card details: use `ErrFailureDetailsUnavailable` when live details
|
|
are gone. Repositories and HTTP both apply the default page limit 100 and
|
|
maximum 500. Cursors remain opaque and are bound to the original query. An empty
|
|
scan page with a next cursor does not signal completion.
|
|
|
|
Archive discovery, scheduling, snapshot loading, and result registration use
|
|
separate narrow interfaces. Registration is conditional on an opaque work
|
|
version. Archives contain run and round aggregates only. Archive receipts use
|
|
a schema version, storage-owned path, lowercase SHA-256 checksum of stored
|
|
bytes, and publication time. Archive schema version starts at 1.
|
|
|
|
Replay acceptance atomically reserves `(sourceRunId, idempotencyKey)` with a
|
|
SHA-256 fingerprint of the source and exact requestedBy body value. Repeating
|
|
the request returns its original action/target; changing the request returns
|
|
`ErrConflict`. The action fixes its target ID before target creation/enqueue.
|
|
|
|
Replay work discovery, lease management, progress updates, source pagination,
|
|
target creation/enqueue, and source cleanup are separate interfaces. Progress
|
|
updates are fenced and use a revision check. Target enqueue distinguishes
|
|
durably present duplicates from rejected writes; a zero insert count by itself
|
|
does not authorize source cleanup. Cleanup checks exact source record versions
|
|
and target acceptance evidence. Implementations must persist enough evidence
|
|
to recover partial batches; a scan cursor by itself is insufficient. Source
|
|
pagination must remain correct while already-enqueued entries are deleted.
|
|
|
|
`ErrNotFound`, `ErrConflict`, `ErrInvalidArgument`,
|
|
`ErrFailureDetailsUnavailable`, `ErrLeaseLost`, and `ErrBusy` support
|
|
`errors.Is` through wrapping. HTTP error envelopes and status mappings remain
|
|
the HTTP implementation's responsibility.
|
|
|
|
## Redis naming
|
|
|
|
All names are prefixed with the configured namespace (`flow:` by default;
|
|
`sim-status:` for status sync). Existing run/round/task-state/failure names are
|
|
preserved. New keys include:
|
|
|
|
| Suffix after namespace | Scope |
|
|
| --- | --- |
|
|
| `{runId}:round:N:stats` | Run slot |
|
|
| `{runId}:archive` | Run slot |
|
|
| `runs:all` | Global index |
|
|
| `archive:pending` | Global index |
|
|
| `{replay}:action:actionId:meta` | Replay slot |
|
|
| `{replay}:action:actionId:lease` | Replay slot |
|
|
| `{replay}:idempotency:SHA256(sourceRunId,key)` | Replay slot; tuple encoded as a JSON array before hashing |
|
|
| `{replay}:pending` | Replay slot |
|
|
|
|
Replay action creation, idempotency, and discovery can be atomic within their
|
|
shared slot. Source/target runs and global indexes can occupy different slots;
|
|
their coordination requires recoverable registrations/receipts rather than a
|
|
single cross-slot Lua transaction. New identifier validation rejects hash-tag
|
|
or path delimiters without rewriting legacy JSON on decode.
|
|
|
|
## HTTP surface
|
|
|
|
| Method | Route | Response |
|
|
| --- | --- | --- |
|
|
| GET | `/internal/v1/full-sync-runs?syncKind=&limit=` | `RunList` merged from active Redis Runs and `run-<runId>.json` archives |
|
|
| GET | `/internal/v1/full-sync-runs/{runId}` | `RunView` with `source`, `run`, optional `rounds` |
|
|
| GET | `/internal/v1/full-sync-runs/{runId}/failures?cursor=&limit=` | `FailurePage` with `items`, `nextCursor` |
|
|
| POST | `/internal/v1/full-sync-runs/{runId}/replay` | HTTP 202 `ReplayAccepted` |
|
|
| GET | `/internal/v1/full-sync-actions/{actionId}` | `ReplayAction` |
|
|
|
|
POST requires `Idempotency-Key` and JSON `{"requestedBy":"..."}`. Authentication,
|
|
HTTP handlers, storage, replay execution, archive files, and service wiring are
|
|
intentionally left to subsequent implementations.
|
|
|
|
## Verification
|
|
|
|
```text
|
|
gofmt -l internal/fullsync/contract
|
|
go test ./internal/fullsync/contract
|
|
go test ./internal/queue/zset ./internal/consumer/zset ./internal/producer/simstatus ./internal/producer/monthlyusage ./internal/service/simstatus ./internal/service/monthlyusage
|
|
```
|
|
|
|
Tests cover legacy schema and JSON compatibility, existing key compatibility,
|
|
cluster key boundaries, bounded categories/messages, overflow, run accounting,
|
|
pagination limits, status domains, replay identity, internal field exclusion,
|
|
and receipt/progress validation. Regression tests use the repository's isolated
|
|
Redis helper and require `redis-server` or `REDIS_SERVER_BINARY`.
|