chore: 并入 iotlcm-worker-svr 历史到 worker-svr/（原提交 96f0f4e）

worker-svr/internal/fullsync/httpapi/README.md

@@ -0,0 +1,99 @@
+# Full-sync HTTP control API
+
+The handler imports no queue or storage implementations. Inject the five
+contract interfaces in `Dependencies`: `RunLister`, `RunReader`,
+`FailurePager`, `ReplayRequester`, and `ActionReader`. The replay service owns durable
+idempotency and execution; the HTTP layer does not cache requests or launch work.
+
+Use `NewRunReader(live, archive, rounds)` to compose `LiveRunReader`,
+`ArchivedRunReader`, and `RoundStatsReader` into a `RunReader`. Live metadata
+wins and returns `source: "live"`; only `ErrNotFound` from live metadata permits
+archive fallback, returning `source: "archive"`. Live storage errors and round
+lookup errors propagate. Archive and round readers may be nil to disable those
+lookups. A failure listing always calls `FailurePager` directly, never archives.
+
+## Routes and responses
+
+All routes require `Authorization: Bearer <configured token>`. Missing, wrong,
+or ambiguous credentials return 401 with a `WWW-Authenticate` challenge. An
+empty configured token is a startup configuration error. Responses use JSON and
+`Cache-Control: no-store`.
+
+| Method | Route | Success body |
+| --- | --- | --- |
+| GET | `/internal/v1/full-sync-runs?syncKind=&limit=` | 200 `contract.RunList` |
+| GET | `/internal/v1/full-sync-runs/{runId}` | 200 `contract.RunView` |
+| GET | `/internal/v1/full-sync-runs/{runId}/failures?cursor=&limit=` | 200 `contract.FailurePage` |
+| POST | `/internal/v1/full-sync-runs/{runId}/replay` | 202 `contract.ReplayAccepted` |
+| GET | `/internal/v1/full-sync-actions/{actionId}` | 200 `contract.ReplayAction` |
+
+Run list results merge active Redis metadata with flat `run-<runId>.json`
+archives under the configured archive root. Active Redis metadata wins when an
+ID exists in both sources. `syncKind` accepts `traffic_usage` or `sim_status`;
+`limit` defaults to 50 and is capped at 100.
+
+Failure limits use the contract rules: absent, empty, or zero selects 100;
+positive values up to 500 are accepted; other values are rejected. Repeated
+cursor/limit parameters and malformed query encoding are rejected. Cursors are
+opaque and passed unchanged after URL decoding. The pager validates cursor
+contents and run binding. Empty pages serialize `items: []` and preserve
+`nextCursor`; only an empty next cursor signals completion.
+
+Replay requires a single `Idempotency-Key` header and a JSON body such as
+`{"requestedBy":"operator"}`. The body is limited to 64 KiB, must contain one
+JSON object with no unknown fields, and must satisfy `ReplayRequest.Validate`.
+If supplied, Content-Type must be `application/json` (parameters are allowed).
+The exact `requestedBy` value is preserved for fingerprinting. Repeated accepted
+requests return the service's existing action/target and current status with
+202. The adapter constructs `statusUrl` and the `Location` header from actionId.
+
+All application errors use this envelope, without storage error details:
+
+```json
+{"error":{"code":"invalid_argument","message":"Invalid request"}}
+```
+
+| Condition | HTTP status | Error code |
+| --- | --- | --- |
+| Invalid path, query, header, body, or `ErrInvalidArgument` | 400 | `invalid_argument` |
+| Missing/wrong bearer credentials | 401 | `unauthorized` |
+| Unknown route/resource or `ErrNotFound` | 404 | `not_found` |
+| Unsupported route method | 405 | `method_not_allowed` |
+| `ErrConflict` (including changed idempotent request), `ErrLeaseLost` | 409 | `conflict` |
+| `ErrFailureDetailsUnavailable` | 410 | `failure_details_unavailable` |
+| Oversized replay body | 413 | `body_too_large` |
+| `ErrBusy` | 503 | `busy` |
+| Other service errors or unencodable service output | 500 | `internal_error` |
+
+## Lifecycle
+
+`NewHandler(token, deps)` can be mounted by an existing HTTP server. For an
+independent listener, construct `NewServer(Config{Addr: address, BearerToken:
+token}, deps)` and call `Start(workerCtx)`. `Start` binds synchronously and
+returns address-in-use and other startup errors directly. `Addr()` exposes the
+bound address, including an OS-assigned port when configured with port zero.
+
+Call `Wait()` after Start, normally in the worker's supervision goroutine. It
+returns serving or shutdown failures; it also returns startup failures if Start
+failed. The server is single-use and rejects a second Start. Wait can have
+multiple callers but must not be called before Start.
+
+Canceling `workerCtx` closes the listener and drains active HTTP requests.
+Active request contexts stay usable during the drain. `ShutdownTimeout` defaults
+to ten seconds; if it expires, active connections are closed and Wait returns
+an error wrapping `context.DeadlineExceeded`. Normal cancellation with a
+successful drain returns nil. No worker wiring or configuration-file changes
+are part of this package.
+
+## Verification
+
+```text
+gofmt -l internal/fullsync/httpapi
+go test ./internal/fullsync/...
+go test ./internal/fullsync/httpapi -cover
+go vet ./internal/fullsync/...
+```
+
+Tests use contract mocks and local HTTP listeners, requiring no Redis or archive
+files. `go test -race ./internal/fullsync/...` additionally requires a supported
+C compiler and CGO enabled.

worker-svr/internal/fullsync/httpapi/SIMPLE_REPLAY.md

@@ -0,0 +1,10 @@
+# Simple full-sync replay
+
+The Worker control server listens on `:18081` inside the private Docker network. Do not publish port 18081 to the host or the internet. Admin already enforces user JWT authentication and replay permissions and calls `http://iotlcm-worker-svr:18081` directly; no additional Worker URL or bearer token environment variables are needed.
+
+- `POST /api/v1/iot/fullSyncRuns/replay` with `{ "sourceRunId": "..." }` returns `replayRunId` and `accepted`. Admin requires `cpm:iot:fullSyncRuns:replay`.
+- `GET /api/v1/iot/fullSyncRuns/replay/{replayRunId}` reports admission (`accepted`, `processing`, `needs_attention`, `completed`), remaining source records, enqueued count, and the independent target Run status. Requires `cpm:iot:fullSyncRuns:replay:view`.
+
+All failures in a completed `flow:` or `sim-status:` Run are selected. No retryability filter; the target Run has exactly one business round. Source failure entries are deleted only after target task-state confirms enqueue; original outcome counts and historical JSON remain unchanged. Repeat requests to the same source return the same target. After repairing a `needs_attention` condition, repeat the request to resume it. To retry again after completion, use the new Replay Run as the source.
+
+No deployment or live Replay request is part of this change.

worker-svr/internal/fullsync/httpapi/handler.go

@@ -0,0 +1,321 @@
+// Package httpapi exposes the internal full-sync control API through contract
+// interfaces. Persistence and replay execution belong to the injected services.
+package httpapi
+
+import (
+	"bytes"
+	"crypto/subtle"
+	"encoding/json"
+	"errors"
+	"io"
+	"mime"
+	"net/http"
+	"net/url"
+	"strconv"
+	"strings"
+	"unicode"
+	"unicode/utf8"
+
+	"git.5iot.com/iotlcm-public/iotlcm-worker-svr/internal/fullsync/contract"
+	fullsyncruntime "git.5iot.com/iotlcm-public/iotlcm-worker-svr/internal/fullsync/runtime"
+
+	"github.com/gogf/gf/v2/errors/gerror"
+)
+
+const maxReplayBodyBytes = 64 << 10
+
+// Dependencies are the only application services used by the HTTP handlers.
+type Dependencies struct {
+	Runs     contract.RunReader
+	List     contract.RunLister
+	Failures contract.FailurePager
+	Replays  contract.ReplayRequester
+	Actions  contract.ActionReader
+	Simple   *fullsyncruntime.SimpleReplay
+}
+
+// ErrorResponse is shared by authentication, routing, validation and service
+// failures. Messages deliberately exclude underlying storage error details.
+type ErrorResponse struct {
+	Error APIError `json:"error"`
+}
+
+type APIError struct {
+	Code    string `json:"code"`
+	Message string `json:"message"`
+}
+
+// NewHandler exposes the control routes on the private Docker network. Admin
+// performs the user JWT and permission checks before calling this listener.
+func NewHandler(bearerToken string, deps Dependencies) (http.Handler, error) {
+	if strings.ContainsFunc(bearerToken, func(r rune) bool { return unicode.IsSpace(r) || unicode.IsControl(r) }) {
+		return nil, gerror.New("invalid full-sync HTTP token")
+	}
+	if deps.Runs == nil || deps.List == nil || deps.Failures == nil || deps.Replays == nil || deps.Actions == nil {
+		return nil, gerror.New("full-sync HTTP requires run list, run, failure, replay and action services")
+	}
+	mux := http.NewServeMux()
+	mux.HandleFunc(contract.ListRunsRoute, method(http.MethodGet, func(w http.ResponseWriter, r *http.Request) {
+		query, err := url.ParseQuery(r.URL.RawQuery)
+		if err != nil || len(query["limit"]) > 1 || len(query["syncKind"]) > 1 {
+			writeError(w, contract.ErrInvalidArgument)
+			return
+		}
+		filter := contract.RunListFilter{SyncKind: contract.SyncKind(query.Get("syncKind"))}
+		if filter.SyncKind != "" && !filter.SyncKind.Valid() {
+			writeError(w, contract.ErrInvalidArgument)
+			return
+		}
+		if query.Get("limit") != "" {
+			filter.Limit, err = strconv.Atoi(query.Get("limit"))
+			if err != nil || filter.Limit < 1 || filter.Limit > 100 {
+				writeError(w, contract.ErrInvalidArgument)
+				return
+			}
+		}
+		result, err := deps.List.ListRuns(r.Context(), filter)
+		if err != nil {
+			writeError(w, err)
+			return
+		}
+		writeJSON(w, http.StatusOK, result)
+	}))
+	mux.HandleFunc(contract.GetRunRoute, method(http.MethodGet, func(w http.ResponseWriter, r *http.Request) {
+		id := r.PathValue("runId")
+		if err := contract.ValidateID(id); err != nil {
+			writeError(w, err)
+			return
+		}
+		view, err := deps.Runs.GetRun(r.Context(), id)
+		if err != nil {
+			writeError(w, err)
+			return
+		}
+		writeJSON(w, http.StatusOK, view)
+	}))
+	mux.HandleFunc(contract.GetFailuresRoute, method(http.MethodGet, func(w http.ResponseWriter, r *http.Request) {
+		id := r.PathValue("runId")
+		if err := contract.ValidateID(id); err != nil {
+			writeError(w, err)
+			return
+		}
+		page, err := parsePage(r.URL.RawQuery)
+		if err != nil {
+			writeError(w, err)
+			return
+		}
+		result, err := deps.Failures.ListFailures(r.Context(), id, page)
+		if err != nil {
+			writeError(w, err)
+			return
+		}
+		if result.Items == nil {
+			result.Items = []contract.Failure{}
+		}
+		writeJSON(w, http.StatusOK, result)
+	}))
+	mux.HandleFunc(contract.PostReplayRoute, method(http.MethodPost, func(w http.ResponseWriter, r *http.Request) {
+		if deps.Simple != nil {
+			id := r.PathValue("runId")
+			if err := contract.ValidateID(id); err != nil {
+				writeError(w, err)
+				return
+			}
+			var request struct {
+				RequestedBy string `json:"requestedBy"`
+				MaxRounds   int    `json:"maxRounds"`
+			}
+			decoder := json.NewDecoder(http.MaxBytesReader(w, r.Body, maxReplayBodyBytes))
+			decoder.DisallowUnknownFields()
+			if err := decoder.Decode(&request); err != nil {
+				writeError(w, contract.ErrInvalidArgument)
+				return
+			}
+			if err := decoder.Decode(new(any)); err != io.EOF {
+				writeError(w, contract.ErrInvalidArgument)
+				return
+			}
+			accepted, err := deps.Simple.Request(r.Context(), id, request.RequestedBy, request.MaxRounds)
+			if errors.Is(err, fullsyncruntime.ErrNoFailures) {
+				writeAPIError(w, http.StatusUnprocessableEntity, "no_failures", "Run has no failures to replay")
+				return
+			}
+			if err != nil {
+				writeError(w, err)
+				return
+			}
+			writeJSON(w, http.StatusAccepted, accepted)
+			return
+		}
+		request, err := parseReplay(w, r)
+		if err != nil {
+			var tooLarge *http.MaxBytesError
+			if errors.As(err, &tooLarge) {
+				writeAPIError(w, http.StatusRequestEntityTooLarge, "body_too_large", "Replay request body is too large")
+			} else {
+				writeError(w, err)
+			}
+			return
+		}
+		accepted, err := deps.Replays.RequestReplay(r.Context(), request)
+		if err != nil {
+			writeError(w, err)
+			return
+		}
+		// Construct a local status URL from the returned identity, never an
+		// externally supplied redirect. Invalid service output is a server error.
+		accepted.StatusURL, err = contract.ActionStatusURL(accepted.ActionID)
+		if err != nil || contract.ValidateID(accepted.TargetRunID) != nil || !accepted.Status.Valid() {
+			writeAPIError(w, http.StatusInternalServerError, "internal_error", "Internal server error")
+			return
+		}
+		w.Header().Set("Location", accepted.StatusURL)
+		writeJSON(w, http.StatusAccepted, accepted)
+	}))
+	mux.HandleFunc("/internal/v1/full-sync-replays/{replayRunId}", method(http.MethodGet, func(w http.ResponseWriter, r *http.Request) {
+		if deps.Simple == nil {
+			writeError(w, contract.ErrNotFound)
+			return
+		}
+		status, err := deps.Simple.Status(r.Context(), r.PathValue("replayRunId"))
+		if err != nil {
+			writeError(w, err)
+			return
+		}
+		writeJSON(w, http.StatusOK, status)
+	}))
+	mux.HandleFunc(contract.GetActionRoute, method(http.MethodGet, func(w http.ResponseWriter, r *http.Request) {
+		if deps.Simple != nil {
+			writeError(w, contract.ErrNotFound)
+			return
+		}
+		id := r.PathValue("actionId")
+		if err := contract.ValidateID(id); err != nil {
+			writeError(w, err)
+			return
+		}
+		action, err := deps.Actions.GetReplayAction(r.Context(), id)
+		if err != nil {
+			writeError(w, err)
+			return
+		}
+		writeJSON(w, http.StatusOK, action)
+	}))
+	mux.HandleFunc("/", func(w http.ResponseWriter, _ *http.Request) {
+		writeError(w, contract.ErrNotFound)
+	})
+	if bearerToken == "" {
+		return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+			w.Header().Set("Cache-Control", "no-store")
+			mux.ServeHTTP(w, r)
+		}), nil
+	}
+	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.Header().Set("Cache-Control", "no-store")
+		values := r.Header.Values("Authorization")
+		if len(values) == 1 {
+			parts := strings.Fields(values[0])
+			if len(parts) == 2 && strings.EqualFold(parts[0], "Bearer") && subtle.ConstantTimeCompare([]byte(parts[1]), []byte(bearerToken)) == 1 {
+				mux.ServeHTTP(w, r)
+				return
+			}
+		}
+		w.Header().Set("WWW-Authenticate", `Bearer realm="full-sync"`)
+		writeAPIError(w, http.StatusUnauthorized, "unauthorized", "Valid bearer authentication is required")
+	}), nil
+}
+
+func method(expected string, next http.HandlerFunc) http.HandlerFunc {
+	return func(w http.ResponseWriter, r *http.Request) {
+		if r.Method != expected {
+			w.Header().Set("Allow", expected)
+			writeAPIError(w, http.StatusMethodNotAllowed, "method_not_allowed", "Method not allowed")
+			return
+		}
+		next(w, r)
+	}
+}
+
+func parsePage(rawQuery string) (contract.PageRequest, error) {
+	query, err := url.ParseQuery(rawQuery)
+	if err != nil || len(query["cursor"]) > 1 || len(query["limit"]) > 1 {
+		return contract.PageRequest{}, gerror.Wrap(contract.ErrInvalidArgument, "invalid pagination query")
+	}
+	page := contract.PageRequest{Cursor: contract.Cursor(query.Get("cursor"))}
+	if value := query.Get("limit"); value != "" {
+		page.Limit, err = strconv.Atoi(value)
+		if err != nil {
+			return contract.PageRequest{}, gerror.Wrap(contract.ErrInvalidArgument, "limit must be an integer")
+		}
+	}
+	return page.Normalized()
+}
+
+func parseReplay(w http.ResponseWriter, r *http.Request) (contract.ReplayRequest, error) {
+	var request contract.ReplayRequest
+	if value := r.Header.Get("Content-Type"); value != "" {
+		mediaType, _, err := mime.ParseMediaType(value)
+		if err != nil || mediaType != "application/json" {
+			return request, gerror.Wrap(contract.ErrInvalidArgument, "replay body must be JSON")
+		}
+	}
+	data, err := io.ReadAll(http.MaxBytesReader(w, r.Body, maxReplayBodyBytes))
+	if err != nil {
+		var tooLarge *http.MaxBytesError
+		if errors.As(err, &tooLarge) {
+			return request, err
+		}
+		return request, gerror.Wrap(contract.ErrInvalidArgument, "cannot read replay body")
+	}
+	if !utf8.Valid(data) {
+		return request, gerror.Wrap(contract.ErrInvalidArgument, "replay body must be UTF-8")
+	}
+	decoder := json.NewDecoder(bytes.NewReader(data))
+	decoder.DisallowUnknownFields()
+	if err := decoder.Decode(&request); err != nil {
+		return request, gerror.Wrap(contract.ErrInvalidArgument, "invalid replay JSON")
+	}
+	if err := decoder.Decode(new(any)); err != io.EOF {
+		return request, gerror.Wrap(contract.ErrInvalidArgument, "replay body must contain exactly one JSON object")
+	}
+	if len(r.Header.Values(contract.IdempotencyHeader)) != 1 {
+		return request, gerror.Wrap(contract.ErrInvalidArgument, "exactly one Idempotency-Key is required")
+	}
+	request.SourceRunID = r.PathValue("runId")
+	request.IdempotencyKey = r.Header.Get(contract.IdempotencyHeader)
+	return request, request.Validate()
+}
+
+func writeError(w http.ResponseWriter, err error) {
+	switch {
+	case errors.Is(err, contract.ErrInvalidArgument):
+		writeAPIError(w, http.StatusBadRequest, "invalid_argument", "Invalid request")
+	case errors.Is(err, contract.ErrNotFound):
+		writeAPIError(w, http.StatusNotFound, "not_found", "Resource not found")
+	case errors.Is(err, contract.ErrFailureDetailsUnavailable):
+		writeAPIError(w, http.StatusGone, "failure_details_unavailable", "Live failure details are no longer available")
+	case errors.Is(err, contract.ErrConflict), errors.Is(err, contract.ErrLeaseLost):
+		writeAPIError(w, http.StatusConflict, "conflict", "Request conflicts with the current resource")
+	case errors.Is(err, contract.ErrBusy):
+		writeAPIError(w, http.StatusServiceUnavailable, "busy", "Resource is busy; retry later")
+	default:
+		writeAPIError(w, http.StatusInternalServerError, "internal_error", "Internal server error")
+	}
+}
+
+func writeAPIError(w http.ResponseWriter, status int, code, message string) {
+	writeJSON(w, status, ErrorResponse{Error: APIError{Code: code, Message: message}})
+}
+
+func writeJSON(w http.ResponseWriter, status int, value any) {
+	// Marshal before writing headers so an invalid service payload cannot
+	// produce a successful status with truncated JSON.
+	data, err := json.Marshal(value)
+	if err != nil {
+		status = http.StatusInternalServerError
+		data = []byte(`{"error":{"code":"internal_error","message":"Internal server error"}}`)
+	}
+	w.Header().Set("Content-Type", "application/json; charset=utf-8")
+	w.WriteHeader(status)
+	_, _ = w.Write(append(data, '\n'))
+}

worker-svr/internal/fullsync/httpapi/handler_test.go

@@ -0,0 +1,437 @@
+package httpapi
+
+import (
+	"context"
+	"encoding/json"
+	"fmt"
+	"net/http"
+	"net/http/httptest"
+	"net/url"
+	"reflect"
+	"strings"
+	"testing"
+
+	"git.5iot.com/iotlcm-public/iotlcm-worker-svr/internal/fullsync/contract"
+)
+
+const (
+	testToken   = "test-secret"
+	runPath     = "/internal/v1/full-sync-runs/run-1"
+	failurePath = runPath + "/failures"
+	replayPath  = runPath + "/replay"
+	actionPath  = "/internal/v1/full-sync-actions/action-1"
+)
+
+type runFunc func(context.Context, string) (contract.RunView, error)
+
+func (f runFunc) GetRun(ctx context.Context, id string) (contract.RunView, error) {
+	return f(ctx, id)
+}
+
+type runListFunc func(context.Context, contract.RunListFilter) (contract.RunList, error)
+
+func (f runListFunc) ListRuns(ctx context.Context, filter contract.RunListFilter) (contract.RunList, error) {
+	return f(ctx, filter)
+}
+
+type failureFunc func(context.Context, string, contract.PageRequest) (contract.FailurePage, error)
+
+func (f failureFunc) ListFailures(ctx context.Context, id string, page contract.PageRequest) (contract.FailurePage, error) {
+	return f(ctx, id, page)
+}
+
+type replayFunc func(context.Context, contract.ReplayRequest) (contract.ReplayAccepted, error)
+
+func (f replayFunc) RequestReplay(ctx context.Context, req contract.ReplayRequest) (contract.ReplayAccepted, error) {
+	return f(ctx, req)
+}
+
+type actionFunc func(context.Context, string) (contract.ReplayAction, error)
+
+func (f actionFunc) GetReplayAction(ctx context.Context, id string) (contract.ReplayAction, error) {
+	return f(ctx, id)
+}
+
+func testDependencies() Dependencies {
+	return Dependencies{
+		Runs: runFunc(func(context.Context, string) (contract.RunView, error) {
+			return contract.RunView{}, contract.ErrNotFound
+		}),
+		List: runListFunc(func(context.Context, contract.RunListFilter) (contract.RunList, error) {
+			return contract.RunList{}, contract.ErrNotFound
+		}),
+		Failures: failureFunc(func(context.Context, string, contract.PageRequest) (contract.FailurePage, error) {
+			return contract.FailurePage{}, contract.ErrNotFound
+		}),
+		Replays: replayFunc(func(context.Context, contract.ReplayRequest) (contract.ReplayAccepted, error) {
+			return contract.ReplayAccepted{}, contract.ErrNotFound
+		}),
+		Actions: actionFunc(func(context.Context, string) (contract.ReplayAction, error) {
+			return contract.ReplayAction{}, contract.ErrNotFound
+		}),
+	}
+}
+
+func testHandler(t *testing.T, deps Dependencies) http.Handler {
+	t.Helper()
+	handler, err := NewHandler(testToken, deps)
+	if err != nil {
+		t.Fatal(err)
+	}
+	return handler
+}
+
+func request(handler http.Handler, method, path, body string, headers ...string) *httptest.ResponseRecorder {
+	req := httptest.NewRequest(method, path, strings.NewReader(body))
+	req.Header.Set("Authorization", "Bearer "+testToken)
+	for i := 0; i < len(headers); i += 2 {
+		req.Header.Set(headers[i], headers[i+1])
+	}
+	recorder := httptest.NewRecorder()
+	handler.ServeHTTP(recorder, req)
+	return recorder
+}
+
+func assertError(t *testing.T, response *httptest.ResponseRecorder, status int, code string) {
+	t.Helper()
+	if response.Code != status {
+		t.Fatalf("status = %d, want %d: %s", response.Code, status, response.Body)
+	}
+	if !strings.HasPrefix(response.Header().Get("Content-Type"), "application/json") {
+		t.Fatalf("missing JSON content type: %v", response.Header())
+	}
+	var body ErrorResponse
+	if err := json.Unmarshal(response.Body.Bytes(), &body); err != nil {
+		t.Fatal(err)
+	}
+	if body.Error.Code != code || body.Error.Message == "" {
+		t.Fatalf("error = %+v, want code %q and message", body, code)
+	}
+}
+
+func TestAuthenticationProtectsEveryRoute(t *testing.T) {
+	deps := testDependencies()
+	deps.Runs = runFunc(func(context.Context, string) (contract.RunView, error) {
+		t.Fatal("unauthenticated request reached service")
+		return contract.RunView{}, nil
+	})
+	handler := testHandler(t, deps)
+	for _, path := range []string{runPath, failurePath, replayPath, actionPath, "/unknown"} {
+		for _, auth := range []string{"", "Bearer", "Bearer wrong", "Basic " + testToken, "Bearer " + testToken + " extra"} {
+			t.Run(path+"/"+auth, func(t *testing.T) {
+				response := request(handler, http.MethodGet, path, "", "Authorization", auth)
+				assertError(t, response, 401, "unauthorized")
+				if response.Header().Get("WWW-Authenticate") == "" {
+					t.Fatal("missing authentication challenge")
+				}
+			})
+		}
+	}
+	req := httptest.NewRequest(http.MethodGet, runPath, nil)
+	req.Header.Add("Authorization", "Bearer "+testToken)
+	req.Header.Add("Authorization", "Bearer "+testToken)
+	response := httptest.NewRecorder()
+	handler.ServeHTTP(response, req)
+	assertError(t, response, 401, "unauthorized")
+}
+
+func TestRunListQuery(t *testing.T) {
+	deps := testDependencies()
+	deps.List = runListFunc(func(_ context.Context, filter contract.RunListFilter) (contract.RunList, error) {
+		if filter.SyncKind != contract.SyncKindTrafficUsage || filter.Limit != 7 {
+			t.Fatalf("filter = %+v", filter)
+		}
+		return contract.RunList{Items: []contract.RunView{{Source: contract.SourceLive, Run: contract.RunMeta{RunID: "run-1", Operation: "current_month_usage_full_sync"}}}, Total: 1}, nil
+	})
+	response := request(testHandler(t, deps), http.MethodGet, "/internal/v1/full-sync-runs?syncKind=traffic_usage&limit=7", "")
+	if response.Code != http.StatusOK || !strings.Contains(response.Body.String(), "run-1") {
+		t.Fatalf("response = %d %s", response.Code, response.Body)
+	}
+	for _, query := range []string{"syncKind=invalid", "limit=101", "limit=abc", "limit=1&limit=2"} {
+		assertError(t, request(testHandler(t, deps), http.MethodGet, "/internal/v1/full-sync-runs?"+query, ""), 400, "invalid_argument")
+	}
+}
+
+func TestRoutesAndValidation(t *testing.T) {
+	handler := testHandler(t, testDependencies())
+	for _, route := range []struct{ method, path string }{
+		{http.MethodGet, runPath}, {http.MethodGet, failurePath},
+		{http.MethodPost, replayPath}, {http.MethodGet, actionPath},
+	} {
+		t.Run(route.path, func(t *testing.T) {
+			response := request(handler, route.method, route.path, `{"requestedBy":"operator"}`, contract.IdempotencyHeader, "key")
+			assertError(t, response, 404, "not_found")
+			wrongMethod := http.MethodPost
+			if route.method == http.MethodPost {
+				wrongMethod = http.MethodGet
+			}
+			response = request(handler, wrongMethod, route.path, "")
+			assertError(t, response, 405, "method_not_allowed")
+			if response.Header().Get("Allow") != route.method {
+				t.Fatalf("Allow = %q", response.Header().Get("Allow"))
+			}
+		})
+	}
+	for _, path := range []string{"/unknown", "/internal/v1/full-sync-runs", runPath + "/other", runPath + "/"} {
+		assertError(t, request(handler, http.MethodGet, path, ""), 404, "not_found")
+	}
+	for _, path := range []string{
+		"/internal/v1/full-sync-runs/%20run",
+		"/internal/v1/full-sync-runs/a%2Fb/failures",
+		"/internal/v1/full-sync-actions/%7Baction%7D",
+	} {
+		assertError(t, request(handler, http.MethodGet, path, ""), 400, "invalid_argument")
+	}
+	assertError(t, request(handler, http.MethodPost, "/internal/v1/full-sync-runs/a%5Cb/replay", `{"requestedBy":"operator"}`, contract.IdempotencyHeader, "key"), 400, "invalid_argument")
+}
+
+func TestRunAndActionResponsesPropagateContext(t *testing.T) {
+	type contextKey struct{}
+	deps := testDependencies()
+	deps.Runs = runFunc(func(ctx context.Context, id string) (contract.RunView, error) {
+		if id != "run-1" || ctx.Value(contextKey{}) != "request" {
+			t.Fatal("run identity or request context lost")
+		}
+		return contract.RunView{Source: contract.SourceLive, Run: contract.RunMeta{RunID: id}}, nil
+	})
+	deps.Actions = actionFunc(func(ctx context.Context, id string) (contract.ReplayAction, error) {
+		if id != "action-1" || ctx.Value(contextKey{}) != "request" {
+			t.Fatal("action identity or request context lost")
+		}
+		return contract.ReplayAction{ActionID: id, Status: contract.ActionProcessing, IdempotencyKey: "private-key", Fingerprint: "private-fingerprint", LeaseToken: "private-lease", Revision: 99}, nil
+	})
+	handler := testHandler(t, deps)
+	for _, path := range []string{runPath, actionPath} {
+		req := httptest.NewRequest(http.MethodGet, path, nil).WithContext(context.WithValue(context.Background(), contextKey{}, "request"))
+		req.Header.Set("Authorization", "bearer "+testToken)
+		response := httptest.NewRecorder()
+		handler.ServeHTTP(response, req)
+		if response.Code != 200 || response.Header().Get("Cache-Control") != "no-store" {
+			t.Fatalf("response = %d %s", response.Code, response.Body)
+		}
+		if strings.Contains(response.Body.String(), "private-") || strings.Contains(response.Body.String(), "revision") {
+			t.Fatalf("internal fields leaked: %s", response.Body)
+		}
+	}
+}
+
+func TestFailurePagination(t *testing.T) {
+	for _, test := range []struct {
+		query  string
+		limit  int
+		cursor contract.Cursor
+	}{
+		{"", 100, ""}, {"?limit=", 100, ""}, {"?limit=0", 100, ""},
+		{"?limit=1", 1, ""}, {"?limit=500", 500, ""},
+		{"?limit=17&cursor=" + url.QueryEscape("opaque/+ token==:雪"), 17, "opaque/+ token==:雪"},
+	} {
+		t.Run(test.query, func(t *testing.T) {
+			deps := testDependencies()
+			calls := 0
+			deps.Failures = failureFunc(func(_ context.Context, id string, page contract.PageRequest) (contract.FailurePage, error) {
+				calls++
+				if id != "run-1" || page.Limit != test.limit || page.Cursor != test.cursor {
+					t.Fatalf("query = %q %+v", id, page)
+				}
+				return contract.FailurePage{NextCursor: "next:opaque/+="}, nil
+			})
+			response := request(testHandler(t, deps), http.MethodGet, failurePath+test.query, "")
+			if response.Code != 200 || calls != 1 {
+				t.Fatalf("response = %d, calls = %d", response.Code, calls)
+			}
+			var page contract.FailurePage
+			if err := json.Unmarshal(response.Body.Bytes(), &page); err != nil {
+				t.Fatal(err)
+			}
+			if page.Items == nil || len(page.Items) != 0 || page.NextCursor != "next:opaque/+=" {
+				t.Fatalf("empty scan page lost array or continuation: %s", response.Body)
+			}
+		})
+	}
+	deps := testDependencies()
+	deps.Failures = failureFunc(func(context.Context, string, contract.PageRequest) (contract.FailurePage, error) {
+		t.Fatal("invalid pagination reached service")
+		return contract.FailurePage{}, nil
+	})
+	handler := testHandler(t, deps)
+	for _, query := range []string{"limit=-1", "limit=501", "limit=abc", "limit=1.5", "limit=999999999999999999999999", "limit=1&limit=2", "cursor=a&cursor=b", "cursor=%ZZ", "cursor=a;b"} {
+		assertError(t, request(handler, http.MethodGet, failurePath+"?"+query, ""), 400, "invalid_argument")
+	}
+}
+
+func TestFailureCursorValidationAndLiveOnly(t *testing.T) {
+	deps := testDependencies()
+	deps.Runs = runFunc(func(context.Context, string) (contract.RunView, error) {
+		t.Fatal("failure route must not query run summaries or fall back to archives")
+		return contract.RunView{}, nil
+	})
+	deps.Failures = failureFunc(func(_ context.Context, _ string, page contract.PageRequest) (contract.FailurePage, error) {
+		if page.Cursor == "another-run-token" {
+			return contract.FailurePage{}, fmt.Errorf("wrong query binding: %w", contract.ErrInvalidArgument)
+		}
+		return contract.FailurePage{}, fmt.Errorf("expired details: %w", contract.ErrFailureDetailsUnavailable)
+	})
+	handler := testHandler(t, deps)
+	assertError(t, request(handler, http.MethodGet, failurePath+"?cursor=another-run-token", ""), 400, "invalid_argument")
+	assertError(t, request(handler, http.MethodGet, failurePath, ""), 410, "failure_details_unavailable")
+}
+
+func TestReplayValidation(t *testing.T) {
+	deps := testDependencies()
+	deps.Replays = replayFunc(func(context.Context, contract.ReplayRequest) (contract.ReplayAccepted, error) {
+		t.Fatal("invalid replay reached service")
+		return contract.ReplayAccepted{}, nil
+	})
+	handler := testHandler(t, deps)
+	for _, body := range []string{"", "{", "null", "[]", `{}`, `{"requestedBy":""}`, `{"requestedBy":"  "}`, `{"requestedBy":12}`, `{"requestedBy":"a","targetRunId":"injected"}`, `{"requestedBy":"a"} {}`, `{"requestedBy":"a"} trailing`, "{\"requestedBy\":\"\xff\"}"} {
+		t.Run(body, func(t *testing.T) {
+			assertError(t, request(handler, http.MethodPost, replayPath, body, contract.IdempotencyHeader, "key"), 400, "invalid_argument")
+		})
+	}
+	for _, key := range []string{"", " ", strings.Repeat("x", contract.MaxIdempotencyKeyBytes+1), "bad\tkey"} {
+		assertError(t, request(handler, http.MethodPost, replayPath, `{"requestedBy":"operator"}`, contract.IdempotencyHeader, key), 400, "invalid_argument")
+	}
+	assertError(t, request(handler, http.MethodPost, replayPath, `{"requestedBy":"operator"}`), 400, "invalid_argument")
+	assertError(t, request(handler, http.MethodPost, replayPath, `{"requestedBy":"operator"}`, contract.IdempotencyHeader, "key", "Content-Type", "text/plain"), 400, "invalid_argument")
+	assertError(t, request(handler, http.MethodPost, replayPath, `{"requestedBy":"`+strings.Repeat("x", maxReplayBodyBytes)+`"}`, contract.IdempotencyHeader, "key"), 413, "body_too_large")
+	req := httptest.NewRequest(http.MethodPost, replayPath, strings.NewReader(`{"requestedBy":"operator"}`))
+	req.Header.Set("Authorization", "Bearer "+testToken)
+	req.Header.Add(contract.IdempotencyHeader, "key")
+	req.Header.Add(contract.IdempotencyHeader, "key")
+	response := httptest.NewRecorder()
+	handler.ServeHTTP(response, req)
+	assertError(t, response, 400, "invalid_argument")
+}
+
+// The mock owns idempotency; the HTTP adapter must pass exact request identity
+// and return the service's existing action/target even across handler instances.
+func TestReplayAcceptedIdempotencyAndConflict(t *testing.T) {
+	deps := testDependencies()
+	var (
+		stored contract.ReplayRequest
+		calls  int
+	)
+	deps.Replays = replayFunc(func(_ context.Context, req contract.ReplayRequest) (contract.ReplayAccepted, error) {
+		calls++
+		if calls == 1 {
+			stored = req
+		}
+		if req != stored {
+			return contract.ReplayAccepted{}, fmt.Errorf("changed fingerprint: %w", contract.ErrConflict)
+		}
+		return contract.ReplayAccepted{ActionID: "action-1", TargetRunID: "replay-1", Status: contract.ActionAccepted, StatusURL: "https://untrusted.invalid"}, nil
+	})
+	var first contract.ReplayAccepted
+	for i := 0; i < 2; i++ {
+		response := request(testHandler(t, deps), http.MethodPost, replayPath, "{\"requestedBy\":\" operator \"}\n", contract.IdempotencyHeader, "key", "Content-Type", "application/json; charset=utf-8")
+		if response.Code != 202 {
+			t.Fatalf("response = %d %s", response.Code, response.Body)
+		}
+		var accepted contract.ReplayAccepted
+		if err := json.Unmarshal(response.Body.Bytes(), &accepted); err != nil {
+			t.Fatal(err)
+		}
+		if accepted.ActionID != "action-1" || accepted.TargetRunID != "replay-1" || accepted.StatusURL != actionPath || response.Header().Get("Location") != actionPath {
+			t.Fatalf("acceptance = %+v, headers = %v", accepted, response.Header())
+		}
+		if i == 0 {
+			first = accepted
+		} else if !reflect.DeepEqual(first, accepted) {
+			t.Fatalf("repeat changed identity: %+v vs %+v", first, accepted)
+		}
+	}
+	if stored.SourceRunID != "run-1" || stored.RequestedBy != " operator " || stored.IdempotencyKey != "key" {
+		t.Fatalf("request identity changed: %+v", stored)
+	}
+	assertError(t, request(testHandler(t, deps), http.MethodPost, replayPath, `{"requestedBy":"operator"}`, contract.IdempotencyHeader, "key"), 409, "conflict")
+	if calls != 3 {
+		t.Fatalf("replay calls = %d", calls)
+	}
+}
+
+func TestWrappedServiceErrorsUseConsistentEnvelope(t *testing.T) {
+	for _, test := range []struct {
+		err    error
+		status int
+		code   string
+	}{
+		{contract.ErrInvalidArgument, 400, "invalid_argument"},
+		{contract.ErrNotFound, 404, "not_found"},
+		{contract.ErrConflict, 409, "conflict"},
+		{contract.ErrLeaseLost, 409, "conflict"},
+		{contract.ErrFailureDetailsUnavailable, 410, "failure_details_unavailable"},
+		{contract.ErrBusy, 503, "busy"},
+		{fmt.Errorf("private database password"), 500, "internal_error"},
+	} {
+		t.Run(test.code, func(t *testing.T) {
+			err := fmt.Errorf("private database detail: %w", test.err)
+			deps := Dependencies{
+				Runs: runFunc(func(context.Context, string) (contract.RunView, error) { return contract.RunView{}, err }),
+				List: runListFunc(func(context.Context, contract.RunListFilter) (contract.RunList, error) {
+					return contract.RunList{}, err
+				}),
+				Failures: failureFunc(func(context.Context, string, contract.PageRequest) (contract.FailurePage, error) {
+					return contract.FailurePage{}, err
+				}),
+				Replays: replayFunc(func(context.Context, contract.ReplayRequest) (contract.ReplayAccepted, error) {
+					return contract.ReplayAccepted{}, err
+				}),
+				Actions: actionFunc(func(context.Context, string) (contract.ReplayAction, error) { return contract.ReplayAction{}, err }),
+			}
+			handler := testHandler(t, deps)
+			for _, path := range []string{runPath, failurePath, actionPath, replayPath} {
+				method := http.MethodGet
+				if path == replayPath {
+					method = http.MethodPost
+				}
+				response := request(handler, method, path, `{"requestedBy":"operator"}`, contract.IdempotencyHeader, "key")
+				assertError(t, response, test.status, test.code)
+				if strings.Contains(response.Body.String(), "private") {
+					t.Fatal("service details leaked")
+				}
+			}
+		})
+	}
+}
+
+func TestInvalidServiceOutputReturns500(t *testing.T) {
+	deps := testDependencies()
+	deps.Replays = replayFunc(func(context.Context, contract.ReplayRequest) (contract.ReplayAccepted, error) {
+		return contract.ReplayAccepted{ActionID: "bad/id"}, nil
+	})
+	deps.Runs = runFunc(func(context.Context, string) (contract.RunView, error) {
+		return contract.RunView{Rounds: []contract.RoundStats{{ErrorCounts: contract.ErrorCounts{contract.ErrorOther: -1}}}}, nil
+	})
+	handler := testHandler(t, deps)
+	assertError(t, request(handler, http.MethodPost, replayPath, `{"requestedBy":"operator"}`, contract.IdempotencyHeader, "key"), 500, "internal_error")
+	assertError(t, request(handler, http.MethodGet, runPath, ""), 500, "internal_error")
+}
+
+func TestNewHandlerRejectsInvalidConfiguration(t *testing.T) {
+	if _, err := NewHandler("", testDependencies()); err != nil {
+		t.Fatalf("empty token should use Docker-network-only access: %v", err)
+	}
+	for _, token := range []string{" ", "a b", "token\n"} {
+		if _, err := NewHandler(token, testDependencies()); err == nil {
+			t.Errorf("accepted token %q", token)
+		}
+	}
+	for i := 0; i < 5; i++ {
+		deps := testDependencies()
+		switch i {
+		case 0:
+			deps.Runs = nil
+		case 1:
+			deps.Failures = nil
+		case 2:
+			deps.Replays = nil
+		case 3:
+			deps.Actions = nil
+		case 4:
+			deps.List = nil
+		}
+		if _, err := NewHandler(testToken, deps); err == nil {
+			t.Errorf("accepted missing dependency %d", i)
+		}
+	}
+}

worker-svr/internal/fullsync/httpapi/run_reader.go

@@ -0,0 +1,48 @@
+package httpapi
+
+import (
+	"context"
+	"errors"
+
+	"git.5iot.com/iotlcm-public/iotlcm-worker-svr/internal/fullsync/contract"
+
+	"github.com/gogf/gf/v2/errors/gerror"
+)
+
+type runReader struct {
+	live    contract.LiveRunReader
+	archive contract.ArchivedRunReader
+	rounds  contract.RoundStatsReader
+}
+
+// NewRunReader composes live-first run queries without knowing either storage
+// implementation. Archive and rounds are optional (nil disables that lookup).
+// Only absent live metadata permits archive fallback; live round errors do not.
+func NewRunReader(live contract.LiveRunReader, archive contract.ArchivedRunReader, rounds contract.RoundStatsReader) (contract.RunReader, error) {
+	if live == nil {
+		return nil, gerror.New("full-sync run reader requires a live reader")
+	}
+	return &runReader{live: live, archive: archive, rounds: rounds}, nil
+}
+
+func (r *runReader) GetRun(ctx context.Context, runID string) (contract.RunView, error) {
+	if err := contract.ValidateID(runID); err != nil {
+		return contract.RunView{}, err
+	}
+	meta, err := r.live.GetRunMeta(ctx, runID)
+	if err == nil {
+		view := contract.RunView{Source: contract.SourceLive, Run: meta}
+		if r.rounds != nil {
+			view.Rounds, err = r.rounds.GetRoundStats(ctx, runID)
+		}
+		return view, err
+	}
+	if !errors.Is(err, contract.ErrNotFound) || r.archive == nil {
+		return contract.RunView{}, err
+	}
+	archive, err := r.archive.GetArchivedRun(ctx, runID)
+	if err != nil {
+		return contract.RunView{}, err
+	}
+	return contract.RunView{Source: contract.SourceArchive, Run: archive.Run, Rounds: archive.Rounds}, nil
+}

worker-svr/internal/fullsync/httpapi/run_reader_test.go

@@ -0,0 +1,125 @@
+package httpapi
+
+import (
+	"context"
+	"encoding/json"
+	"fmt"
+	"net/http"
+	"testing"
+
+	"git.5iot.com/iotlcm-public/iotlcm-worker-svr/internal/fullsync/contract"
+)
+
+type liveFunc func(context.Context, string) (contract.RunMeta, error)
+
+func (f liveFunc) GetRunMeta(ctx context.Context, id string) (contract.RunMeta, error) {
+	return f(ctx, id)
+}
+
+type archiveFunc func(context.Context, string) (contract.RunArchive, error)
+
+func (f archiveFunc) GetArchivedRun(ctx context.Context, id string) (contract.RunArchive, error) {
+	return f(ctx, id)
+}
+
+type roundsFunc func(context.Context, string) ([]contract.RoundStats, error)
+
+func (f roundsFunc) GetRoundStats(ctx context.Context, id string) ([]contract.RoundStats, error) {
+	return f(ctx, id)
+}
+
+func TestRunLookupLiveFirstAndArchiveFallback(t *testing.T) {
+	for _, test := range []struct {
+		name         string
+		liveErr      error
+		roundsErr    error
+		archiveErr   error
+		wantStatus   int
+		wantSource   contract.RunSource
+		wantArchives int
+		wantRounds   int
+	}{
+		{name: "live", wantStatus: 200, wantSource: contract.SourceLive, wantRounds: 1},
+		{name: "archive", liveErr: fmt.Errorf("expired: %w", contract.ErrNotFound), wantStatus: 200, wantSource: contract.SourceArchive, wantArchives: 1},
+		{name: "unknown", liveErr: contract.ErrNotFound, archiveErr: contract.ErrNotFound, wantStatus: 404, wantArchives: 1},
+		{name: "live failure never falls back", liveErr: fmt.Errorf("live offline"), wantStatus: 500},
+		{name: "round error never falls back", roundsErr: fmt.Errorf("rounds offline"), wantStatus: 500, wantRounds: 1},
+		{name: "missing rounds never fall back", roundsErr: contract.ErrNotFound, wantStatus: 404, wantRounds: 1},
+		{name: "archive error", liveErr: contract.ErrNotFound, archiveErr: fmt.Errorf("archive offline"), wantStatus: 500, wantArchives: 1},
+	} {
+		t.Run(test.name, func(t *testing.T) {
+			var liveCalls, archiveCalls, roundsCalls int
+			reader, err := NewRunReader(
+				liveFunc(func(_ context.Context, id string) (contract.RunMeta, error) {
+					liveCalls++
+					if id != "run-1" {
+						t.Fatalf("live ID = %q", id)
+					}
+					return contract.RunMeta{RunID: id, CurrentRound: 2}, test.liveErr
+				}),
+				archiveFunc(func(_ context.Context, id string) (contract.RunArchive, error) {
+					archiveCalls++
+					if id != "run-1" {
+						t.Fatalf("archive ID = %q", id)
+					}
+					return contract.RunArchive{Run: contract.RunMeta{RunID: id, CurrentRound: 3}, Rounds: []contract.RoundStats{{RunID: id, Round: 3}}}, test.archiveErr
+				}),
+				roundsFunc(func(_ context.Context, id string) ([]contract.RoundStats, error) {
+					roundsCalls++
+					if id != "run-1" {
+						t.Fatalf("rounds ID = %q", id)
+					}
+					return []contract.RoundStats{{RunID: id, Round: 2}}, test.roundsErr
+				}),
+			)
+			if err != nil {
+				t.Fatal(err)
+			}
+			deps := testDependencies()
+			deps.Runs = reader
+			response := request(testHandler(t, deps), http.MethodGet, runPath, "")
+			if response.Code != test.wantStatus {
+				t.Fatalf("response = %d %s", response.Code, response.Body)
+			}
+			if liveCalls != 1 || archiveCalls != test.wantArchives || roundsCalls != test.wantRounds {
+				t.Fatalf("calls: live=%d archive=%d rounds=%d", liveCalls, archiveCalls, roundsCalls)
+			}
+			if response.Code == 200 {
+				var view contract.RunView
+				if err := json.Unmarshal(response.Body.Bytes(), &view); err != nil {
+					t.Fatal(err)
+				}
+				if view.Source != test.wantSource || view.Run.RunID != "run-1" || len(view.Rounds) != 1 || view.Rounds[0].Round != view.Run.CurrentRound {
+					t.Fatalf("view = %+v", view)
+				}
+			}
+		})
+	}
+}
+
+func TestRunReaderOptionalServicesAndValidation(t *testing.T) {
+	if _, err := NewRunReader(nil, nil, nil); err == nil {
+		t.Fatal("accepted nil live reader")
+	}
+	calls := 0
+	reader, err := NewRunReader(liveFunc(func(_ context.Context, id string) (contract.RunMeta, error) {
+		calls++
+		if id == "missing" {
+			return contract.RunMeta{}, contract.ErrNotFound
+		}
+		return contract.RunMeta{RunID: id}, nil
+	}), nil, nil)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if _, err := reader.GetRun(context.Background(), "bad/id"); err == nil || calls != 0 {
+		t.Fatal("invalid ID reached live reader")
+	}
+	view, err := reader.GetRun(context.Background(), "run-1")
+	if err != nil || view.Source != contract.SourceLive || view.Rounds != nil {
+		t.Fatalf("view = %+v, err = %v", view, err)
+	}
+	if _, err := reader.GetRun(context.Background(), "missing"); err != contract.ErrNotFound {
+		t.Fatalf("error = %v", err)
+	}
+}

worker-svr/internal/fullsync/httpapi/server.go

@@ -0,0 +1,119 @@
+package httpapi
+
+import (
+	"context"
+	"errors"
+	"net"
+	"net/http"
+	"sync"
+	"time"
+
+	"github.com/gogf/gf/v2/errors/gerror"
+)
+
+// Config is deliberately independent of worker/config-file wiring. Addr must
+// be explicit; use 127.0.0.1:0 to let the OS allocate a local test port.
+type Config struct {
+	Addr            string
+	BearerToken     string // Optional for standalone callers; production Worker leaves it empty.
+	ShutdownTimeout time.Duration
+}
+
+// Server is single-use. Start reports bind errors synchronously; after a
+// successful Start, Wait reports serve or shutdown errors. Canceling the Start
+// context stops accepting requests and drains active requests up to the timeout.
+type Server struct {
+	server          *http.Server
+	shutdownTimeout time.Duration
+	mu              sync.Mutex
+	started         bool
+	addr            net.Addr
+	done            chan struct{}
+	err             error
+}
+
+func NewServer(config Config, deps Dependencies) (*Server, error) {
+	if config.Addr == "" || config.ShutdownTimeout < 0 {
+		return nil, gerror.New("full-sync HTTP requires an address and a nonnegative shutdown timeout")
+	}
+	handler, err := NewHandler(config.BearerToken, deps)
+	if err != nil {
+		return nil, err
+	}
+	if config.ShutdownTimeout == 0 {
+		config.ShutdownTimeout = 10 * time.Second
+	}
+	return &Server{
+		server: &http.Server{
+			Addr: config.Addr, Handler: handler,
+			ReadHeaderTimeout: 5 * time.Second,
+			ReadTimeout:       30 * time.Second,
+			WriteTimeout:      30 * time.Second,
+			IdleTimeout:       60 * time.Second,
+		},
+		shutdownTimeout: config.ShutdownTimeout,
+		done:            make(chan struct{}),
+	}, nil
+}
+
+func (s *Server) Start(ctx context.Context) error {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	if s.started {
+		return gerror.New("full-sync HTTP server has already been started")
+	}
+	s.started = true
+	if err := ctx.Err(); err != nil {
+		s.err = err
+		close(s.done)
+		return err
+	}
+	listener, err := (&net.ListenConfig{}).Listen(ctx, "tcp", s.server.Addr)
+	if err != nil {
+		s.err = gerror.Wrap(err, "listen for full-sync HTTP")
+		close(s.done)
+		return s.err
+	}
+	s.addr = listener.Addr()
+	go s.run(ctx, listener)
+	return nil
+}
+
+// Addr returns the bound address, or nil before a successful Start.
+func (s *Server) Addr() net.Addr {
+	s.mu.Lock()
+	defer s.mu.Unlock()
+	return s.addr
+}
+
+// Wait may be called concurrently after Start (including a failed Start).
+func (s *Server) Wait() error {
+	<-s.done
+	return s.err
+}
+
+func (s *Server) run(ctx context.Context, listener net.Listener) {
+	serveDone := make(chan error, 1)
+	go func() { serveDone <- s.server.Serve(listener) }()
+	var err error
+	select {
+	case err = <-serveDone:
+		// An unexpected listener failure also needs to drain active requests.
+	case <-ctx.Done():
+	}
+	shutdownCtx, cancel := context.WithTimeout(context.Background(), s.shutdownTimeout)
+	shutdownErr := s.server.Shutdown(shutdownCtx)
+	cancel()
+	if shutdownErr != nil {
+		// Shutdown does not close active connections when its deadline expires.
+		_ = s.server.Close()
+	}
+	if err == nil {
+		err = <-serveDone
+	}
+	if errors.Is(err, http.ErrServerClosed) {
+		err = nil
+	}
+	s.err = errors.Join(err, shutdownErr)
+	close(s.done)
+}

worker-svr/internal/fullsync/httpapi/server_test.go

@@ -0,0 +1,226 @@
+package httpapi
+
+import (
+	"context"
+	"errors"
+	"io"
+	"net"
+	"net/http"
+	"sync"
+	"testing"
+	"time"
+
+	"git.5iot.com/iotlcm-public/iotlcm-worker-svr/internal/fullsync/contract"
+)
+
+func awaitServer(t *testing.T, done <-chan error) error {
+	t.Helper()
+	select {
+	case err := <-done:
+		return err
+	case <-time.After(5 * time.Second):
+		t.Fatal("server did not stop")
+		return nil
+	}
+}
+
+func startTestServer(t *testing.T, deps Dependencies, timeout time.Duration) (*Server, context.CancelFunc, <-chan error) {
+	t.Helper()
+	server, err := NewServer(Config{Addr: "127.0.0.1:0", BearerToken: testToken, ShutdownTimeout: timeout}, deps)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if server.Addr() != nil {
+		t.Fatal("address before Start")
+	}
+	ctx, cancel := context.WithCancel(context.Background())
+	t.Cleanup(cancel)
+	if err := server.Start(ctx); err != nil {
+		t.Fatal(err)
+	}
+	done := make(chan error, 1)
+	go func() { done <- server.Wait() }()
+	return server, cancel, done
+}
+
+func serverRequest(server *Server) (*http.Response, error) {
+	req, err := http.NewRequest(http.MethodGet, "http://"+server.Addr().String()+runPath, nil)
+	if err != nil {
+		return nil, err
+	}
+	req.Header.Set("Authorization", "Bearer "+testToken)
+	client := &http.Client{Timeout: 4 * time.Second}
+	return client.Do(req)
+}
+
+func TestServerStartServeAndCancel(t *testing.T) {
+	server, cancel, done := startTestServer(t, testDependencies(), 0)
+	response, err := serverRequest(server)
+	if err != nil {
+		t.Fatal(err)
+	}
+	_ = response.Body.Close()
+	if response.StatusCode != 404 {
+		t.Fatalf("status = %d", response.StatusCode)
+	}
+	if err := server.Start(context.Background()); err == nil {
+		t.Fatal("duplicate Start succeeded")
+	}
+	cancel()
+	if err := awaitServer(t, done); err != nil {
+		t.Fatal(err)
+	}
+	if err := server.Wait(); err != nil {
+		t.Fatal(err)
+	}
+	connection, err := net.DialTimeout("tcp", server.Addr().String(), time.Second)
+	if err == nil {
+		_ = connection.Close()
+		t.Fatal("listener still accepts after shutdown")
+	}
+}
+
+func TestServerGracefulShutdownDrainsActiveRequest(t *testing.T) {
+	var (
+		entered = make(chan struct{})
+		release = make(chan struct{})
+		once    sync.Once
+	)
+	t.Cleanup(func() { once.Do(func() { close(release) }) })
+	deps := testDependencies()
+	deps.Runs = runFunc(func(ctx context.Context, id string) (contract.RunView, error) {
+		close(entered)
+		select {
+		case <-release:
+			return contract.RunView{Source: contract.SourceLive, Run: contract.RunMeta{RunID: id}}, nil
+		case <-ctx.Done():
+			return contract.RunView{}, ctx.Err()
+		}
+	})
+	server, cancel, done := startTestServer(t, deps, 2*time.Second)
+	requestDone := make(chan error, 1)
+	go func() {
+		response, err := serverRequest(server)
+		if err == nil {
+			_, err = io.ReadAll(response.Body)
+			_ = response.Body.Close()
+			if response.StatusCode != 200 {
+				err = errors.New("active request did not complete successfully")
+			}
+		}
+		requestDone <- err
+	}()
+	select {
+	case <-entered:
+	case <-time.After(3 * time.Second):
+		t.Fatal("request did not enter handler")
+	}
+	cancel()
+	select {
+	case err := <-done:
+		t.Fatalf("shutdown returned before draining: %v", err)
+	case <-time.After(50 * time.Millisecond):
+	}
+	once.Do(func() { close(release) })
+	if err := awaitServer(t, requestDone); err != nil {
+		t.Fatal(err)
+	}
+	if err := awaitServer(t, done); err != nil {
+		t.Fatal(err)
+	}
+}
+
+func TestServerShutdownDeadlineClosesActiveConnections(t *testing.T) {
+	var (
+		entered = make(chan struct{})
+		exited  = make(chan struct{})
+	)
+	deps := testDependencies()
+	deps.Runs = runFunc(func(ctx context.Context, _ string) (contract.RunView, error) {
+		close(entered)
+		<-ctx.Done()
+		close(exited)
+		return contract.RunView{}, ctx.Err()
+	})
+	server, cancel, done := startTestServer(t, deps, 50*time.Millisecond)
+	requestDone := make(chan error, 1)
+	go func() {
+		response, err := serverRequest(server)
+		if response != nil {
+			_ = response.Body.Close()
+		}
+		requestDone <- err
+	}()
+	select {
+	case <-entered:
+	case <-time.After(3 * time.Second):
+		t.Fatal("request did not enter handler")
+	}
+	cancel()
+	if err := awaitServer(t, done); !errors.Is(err, context.DeadlineExceeded) {
+		t.Fatalf("shutdown error = %v", err)
+	}
+	select {
+	case <-exited:
+	case <-time.After(3 * time.Second):
+		t.Fatal("forced close did not cancel request context")
+	}
+	_ = awaitServer(t, requestDone)
+}
+
+func TestServerStartupFailureIsSynchronous(t *testing.T) {
+	listener, err := net.Listen("tcp", "127.0.0.1:0")
+	if err != nil {
+		t.Fatal(err)
+	}
+	t.Cleanup(func() { _ = listener.Close() })
+	server, err := NewServer(Config{Addr: listener.Addr().String(), BearerToken: testToken}, testDependencies())
+	if err != nil {
+		t.Fatal(err)
+	}
+	startErr := server.Start(context.Background())
+	if startErr == nil {
+		t.Fatal("Start did not report occupied port")
+	}
+	if server.Wait() != startErr || server.Addr() != nil {
+		t.Fatal("failed startup state was lost")
+	}
+	var netErr *net.OpError
+	if !errors.As(startErr, &netErr) {
+		t.Fatalf("network error cause lost: %v", startErr)
+	}
+}
+
+func TestServerCanceledBeforeStartAndImmediateCancellation(t *testing.T) {
+	server, err := NewServer(Config{Addr: "127.0.0.1:0", BearerToken: testToken}, testDependencies())
+	if err != nil {
+		t.Fatal(err)
+	}
+	ctx, cancel := context.WithCancel(context.Background())
+	cancel()
+	if err := server.Start(ctx); !errors.Is(err, context.Canceled) {
+		t.Fatalf("Start = %v", err)
+	}
+	if err := server.Wait(); !errors.Is(err, context.Canceled) {
+		t.Fatalf("Wait = %v", err)
+	}
+	// Cancellation can win the race before Serve starts accepting requests.
+	for i := 0; i < 10; i++ {
+		_, cancel, done := startTestServer(t, testDependencies(), time.Second)
+		cancel()
+		if err := awaitServer(t, done); err != nil {
+			t.Fatal(err)
+		}
+	}
+}
+
+func TestNewServerRejectsInvalidConfiguration(t *testing.T) {
+	for _, config := range []Config{
+		{BearerToken: testToken},
+		{Addr: "127.0.0.1:0", BearerToken: testToken, ShutdownTimeout: -time.Second},
+	} {
+		if _, err := NewServer(config, testDependencies()); err == nil {
+			t.Fatalf("accepted config %+v", config)
+		}
+	}
+}

worker-svr/internal/fullsync/httpapi/simple_replay_test.go

@@ -0,0 +1,59 @@
+package httpapi
+
+import (
+	"bytes"
+	"context"
+	"encoding/json"
+	"net/http"
+	"net/http/httptest"
+	"testing"
+	"time"
+
+	"git.5iot.com/iotlcm-public/iotlcm-worker-svr/internal/fullsync/runtime"
+	"git.5iot.com/iotlcm-public/iotlcm-worker-svr/internal/queue/zset"
+	"git.5iot.com/iotlcm-public/iotlcm-worker-svr/internal/testutil"
+)
+
+func TestSimpleReplayHTTPPrivateNetworkAcceptance(t *testing.T) {
+	redis := testutil.Redis(t)
+	store, _ := zset.New(redis, "sim-status:")
+	service, _ := runtime.NewSimpleReplay(store)
+	ctx := context.Background()
+	key, _ := store.RunMetaKey("source")
+	failed, _ := store.FailedKey("source")
+	_, _ = redis.Do(ctx, "HSET", key, "runId", "source", "status", "completed_with_failures", "producerDone", "1", "currentRound", "1")
+	record := zset.FailureRecord{TaskID: "task", CardID: 1, ICCID: "card", ChanKey: "ch", TempKey: "template"}
+	value, _ := json.Marshal(record)
+	_, _ = redis.Do(ctx, "HSET", failed, "task", string(value))
+	deps := testDependencies()
+	deps.Simple = service
+	handler, err := NewHandler("", deps)
+	if err != nil {
+		t.Fatal(err)
+	}
+	req := httptest.NewRequest(http.MethodPost, "/internal/v1/full-sync-runs/source/replay", bytes.NewBufferString(`{"requestedBy":"42"}`))
+	w := httptest.NewRecorder()
+	handler.ServeHTTP(w, req)
+	if w.Code != 202 {
+		t.Fatalf("accepted: %d %s", w.Code, w.Body.String())
+	}
+	var receipt runtime.ReplayReceipt
+	if err := json.Unmarshal(w.Body.Bytes(), &receipt); err != nil || receipt.ReplayRunID == "" {
+		t.Fatalf("receipt: %+v %v", receipt, err)
+	}
+	deadline := time.Now().Add(3 * time.Second)
+	for time.Now().Before(deadline) {
+		w := httptest.NewRecorder()
+		req := httptest.NewRequest(http.MethodGet, "/internal/v1/full-sync-replays/"+receipt.ReplayRunID, nil)
+		handler.ServeHTTP(w, req)
+		if w.Code == 200 {
+			var status runtime.ReplayStatus
+			_ = json.Unmarshal(w.Body.Bytes(), &status)
+			if status.Status == "completed" && status.Enqueued == 1 {
+				return
+			}
+		}
+		time.Sleep(20 * time.Millisecond)
+	}
+	t.Fatal("replay HTTP never completed")
+}