chore: 初始开源发布
iotlcm-worker-svr:异步执行中枢(RabbitMQ 消费者 + 全量同步编排)。 - Go module:iotlcm-worker-svr → git.5iot.com/iotlcm-public/iotlcm-worker-svr(214 文件) - 凭据与内部标识脱敏:MySQL/Redis 凭据、私网与生产 IP、内网域名、 库名 xiujie_admin → iotlcm、哨兵名占位 - 159 处真实手机号 → 合法 11 位编造号(保留号段与后 4 位,维持互异, 不破坏「校验 MSISDN 格式」等测试断言) - 移除内部资产:CI 触发器、Jenkins 触发测试文档、预演门禁与镜像 digest 记录 - Dockerfile 改用公共基础镜像(golang:1.25-alpine → alpine:3.23), 移除写死的 git 凭据 - README 重写(架构/全量同步模型/配置/已知失败),补 Apache-2.0 LICENSE,LF 行尾 + .gitattributes 已知:3 个测试包在源仓同一提交上同样失败(client/upstream、 service/allocate、service/inbound),属既有缺陷,本仓未引入亦未修复。
This commit is contained in:
@@ -0,0 +1,99 @@
|
||||
# Full-sync HTTP control API
|
||||
|
||||
The handler imports no queue or storage implementations. Inject the five
|
||||
contract interfaces in `Dependencies`: `RunLister`, `RunReader`,
|
||||
`FailurePager`, `ReplayRequester`, and `ActionReader`. The replay service owns durable
|
||||
idempotency and execution; the HTTP layer does not cache requests or launch work.
|
||||
|
||||
Use `NewRunReader(live, archive, rounds)` to compose `LiveRunReader`,
|
||||
`ArchivedRunReader`, and `RoundStatsReader` into a `RunReader`. Live metadata
|
||||
wins and returns `source: "live"`; only `ErrNotFound` from live metadata permits
|
||||
archive fallback, returning `source: "archive"`. Live storage errors and round
|
||||
lookup errors propagate. Archive and round readers may be nil to disable those
|
||||
lookups. A failure listing always calls `FailurePager` directly, never archives.
|
||||
|
||||
## Routes and responses
|
||||
|
||||
All routes require `Authorization: Bearer <configured token>`. Missing, wrong,
|
||||
or ambiguous credentials return 401 with a `WWW-Authenticate` challenge. An
|
||||
empty configured token is a startup configuration error. Responses use JSON and
|
||||
`Cache-Control: no-store`.
|
||||
|
||||
| Method | Route | Success body |
|
||||
| --- | --- | --- |
|
||||
| GET | `/internal/v1/full-sync-runs?syncKind=&limit=` | 200 `contract.RunList` |
|
||||
| GET | `/internal/v1/full-sync-runs/{runId}` | 200 `contract.RunView` |
|
||||
| GET | `/internal/v1/full-sync-runs/{runId}/failures?cursor=&limit=` | 200 `contract.FailurePage` |
|
||||
| POST | `/internal/v1/full-sync-runs/{runId}/replay` | 202 `contract.ReplayAccepted` |
|
||||
| GET | `/internal/v1/full-sync-actions/{actionId}` | 200 `contract.ReplayAction` |
|
||||
|
||||
Run list results merge active Redis metadata with flat `run-<runId>.json`
|
||||
archives under the configured archive root. Active Redis metadata wins when an
|
||||
ID exists in both sources. `syncKind` accepts `traffic_usage` or `sim_status`;
|
||||
`limit` defaults to 50 and is capped at 100.
|
||||
|
||||
Failure limits use the contract rules: absent, empty, or zero selects 100;
|
||||
positive values up to 500 are accepted; other values are rejected. Repeated
|
||||
cursor/limit parameters and malformed query encoding are rejected. Cursors are
|
||||
opaque and passed unchanged after URL decoding. The pager validates cursor
|
||||
contents and run binding. Empty pages serialize `items: []` and preserve
|
||||
`nextCursor`; only an empty next cursor signals completion.
|
||||
|
||||
Replay requires a single `Idempotency-Key` header and a JSON body such as
|
||||
`{"requestedBy":"operator"}`. The body is limited to 64 KiB, must contain one
|
||||
JSON object with no unknown fields, and must satisfy `ReplayRequest.Validate`.
|
||||
If supplied, Content-Type must be `application/json` (parameters are allowed).
|
||||
The exact `requestedBy` value is preserved for fingerprinting. Repeated accepted
|
||||
requests return the service's existing action/target and current status with
|
||||
202. The adapter constructs `statusUrl` and the `Location` header from actionId.
|
||||
|
||||
All application errors use this envelope, without storage error details:
|
||||
|
||||
```json
|
||||
{"error":{"code":"invalid_argument","message":"Invalid request"}}
|
||||
```
|
||||
|
||||
| Condition | HTTP status | Error code |
|
||||
| --- | --- | --- |
|
||||
| Invalid path, query, header, body, or `ErrInvalidArgument` | 400 | `invalid_argument` |
|
||||
| Missing/wrong bearer credentials | 401 | `unauthorized` |
|
||||
| Unknown route/resource or `ErrNotFound` | 404 | `not_found` |
|
||||
| Unsupported route method | 405 | `method_not_allowed` |
|
||||
| `ErrConflict` (including changed idempotent request), `ErrLeaseLost` | 409 | `conflict` |
|
||||
| `ErrFailureDetailsUnavailable` | 410 | `failure_details_unavailable` |
|
||||
| Oversized replay body | 413 | `body_too_large` |
|
||||
| `ErrBusy` | 503 | `busy` |
|
||||
| Other service errors or unencodable service output | 500 | `internal_error` |
|
||||
|
||||
## Lifecycle
|
||||
|
||||
`NewHandler(token, deps)` can be mounted by an existing HTTP server. For an
|
||||
independent listener, construct `NewServer(Config{Addr: address, BearerToken:
|
||||
token}, deps)` and call `Start(workerCtx)`. `Start` binds synchronously and
|
||||
returns address-in-use and other startup errors directly. `Addr()` exposes the
|
||||
bound address, including an OS-assigned port when configured with port zero.
|
||||
|
||||
Call `Wait()` after Start, normally in the worker's supervision goroutine. It
|
||||
returns serving or shutdown failures; it also returns startup failures if Start
|
||||
failed. The server is single-use and rejects a second Start. Wait can have
|
||||
multiple callers but must not be called before Start.
|
||||
|
||||
Canceling `workerCtx` closes the listener and drains active HTTP requests.
|
||||
Active request contexts stay usable during the drain. `ShutdownTimeout` defaults
|
||||
to ten seconds; if it expires, active connections are closed and Wait returns
|
||||
an error wrapping `context.DeadlineExceeded`. Normal cancellation with a
|
||||
successful drain returns nil. No worker wiring or configuration-file changes
|
||||
are part of this package.
|
||||
|
||||
## Verification
|
||||
|
||||
```text
|
||||
gofmt -l internal/fullsync/httpapi
|
||||
go test ./internal/fullsync/...
|
||||
go test ./internal/fullsync/httpapi -cover
|
||||
go vet ./internal/fullsync/...
|
||||
```
|
||||
|
||||
Tests use contract mocks and local HTTP listeners, requiring no Redis or archive
|
||||
files. `go test -race ./internal/fullsync/...` additionally requires a supported
|
||||
C compiler and CGO enabled.
|
||||
@@ -0,0 +1,10 @@
|
||||
# Simple full-sync replay
|
||||
|
||||
The Worker control server listens on `:18081` inside the private Docker network. Do not publish port 18081 to the host or the internet. Admin already enforces user JWT authentication and replay permissions and calls `http://iotlcm-worker-svr:18081` directly; no additional Worker URL or bearer token environment variables are needed.
|
||||
|
||||
- `POST /api/v1/iot/fullSyncRuns/replay` with `{ "sourceRunId": "..." }` returns `replayRunId` and `accepted`. Admin requires `cpm:iot:fullSyncRuns:replay`.
|
||||
- `GET /api/v1/iot/fullSyncRuns/replay/{replayRunId}` reports admission (`accepted`, `processing`, `needs_attention`, `completed`), remaining source records, enqueued count, and the independent target Run status. Requires `cpm:iot:fullSyncRuns:replay:view`.
|
||||
|
||||
All failures in a completed `flow:` or `sim-status:` Run are selected. No retryability filter; the target Run has exactly one business round. Source failure entries are deleted only after target task-state confirms enqueue; original outcome counts and historical JSON remain unchanged. Repeat requests to the same source return the same target. After repairing a `needs_attention` condition, repeat the request to resume it. To retry again after completion, use the new Replay Run as the source.
|
||||
|
||||
No deployment or live Replay request is part of this change.
|
||||
@@ -0,0 +1,321 @@
|
||||
// Package httpapi exposes the internal full-sync control API through contract
|
||||
// interfaces. Persistence and replay execution belong to the injected services.
|
||||
package httpapi
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"crypto/subtle"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"mime"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strconv"
|
||||
"strings"
|
||||
"unicode"
|
||||
"unicode/utf8"
|
||||
|
||||
"git.5iot.com/iotlcm-public/iotlcm-worker-svr/internal/fullsync/contract"
|
||||
fullsyncruntime "git.5iot.com/iotlcm-public/iotlcm-worker-svr/internal/fullsync/runtime"
|
||||
|
||||
"github.com/gogf/gf/v2/errors/gerror"
|
||||
)
|
||||
|
||||
const maxReplayBodyBytes = 64 << 10
|
||||
|
||||
// Dependencies are the only application services used by the HTTP handlers.
|
||||
type Dependencies struct {
|
||||
Runs contract.RunReader
|
||||
List contract.RunLister
|
||||
Failures contract.FailurePager
|
||||
Replays contract.ReplayRequester
|
||||
Actions contract.ActionReader
|
||||
Simple *fullsyncruntime.SimpleReplay
|
||||
}
|
||||
|
||||
// ErrorResponse is shared by authentication, routing, validation and service
|
||||
// failures. Messages deliberately exclude underlying storage error details.
|
||||
type ErrorResponse struct {
|
||||
Error APIError `json:"error"`
|
||||
}
|
||||
|
||||
type APIError struct {
|
||||
Code string `json:"code"`
|
||||
Message string `json:"message"`
|
||||
}
|
||||
|
||||
// NewHandler exposes the control routes on the private Docker network. Admin
|
||||
// performs the user JWT and permission checks before calling this listener.
|
||||
func NewHandler(bearerToken string, deps Dependencies) (http.Handler, error) {
|
||||
if strings.ContainsFunc(bearerToken, func(r rune) bool { return unicode.IsSpace(r) || unicode.IsControl(r) }) {
|
||||
return nil, gerror.New("invalid full-sync HTTP token")
|
||||
}
|
||||
if deps.Runs == nil || deps.List == nil || deps.Failures == nil || deps.Replays == nil || deps.Actions == nil {
|
||||
return nil, gerror.New("full-sync HTTP requires run list, run, failure, replay and action services")
|
||||
}
|
||||
mux := http.NewServeMux()
|
||||
mux.HandleFunc(contract.ListRunsRoute, method(http.MethodGet, func(w http.ResponseWriter, r *http.Request) {
|
||||
query, err := url.ParseQuery(r.URL.RawQuery)
|
||||
if err != nil || len(query["limit"]) > 1 || len(query["syncKind"]) > 1 {
|
||||
writeError(w, contract.ErrInvalidArgument)
|
||||
return
|
||||
}
|
||||
filter := contract.RunListFilter{SyncKind: contract.SyncKind(query.Get("syncKind"))}
|
||||
if filter.SyncKind != "" && !filter.SyncKind.Valid() {
|
||||
writeError(w, contract.ErrInvalidArgument)
|
||||
return
|
||||
}
|
||||
if query.Get("limit") != "" {
|
||||
filter.Limit, err = strconv.Atoi(query.Get("limit"))
|
||||
if err != nil || filter.Limit < 1 || filter.Limit > 100 {
|
||||
writeError(w, contract.ErrInvalidArgument)
|
||||
return
|
||||
}
|
||||
}
|
||||
result, err := deps.List.ListRuns(r.Context(), filter)
|
||||
if err != nil {
|
||||
writeError(w, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, result)
|
||||
}))
|
||||
mux.HandleFunc(contract.GetRunRoute, method(http.MethodGet, func(w http.ResponseWriter, r *http.Request) {
|
||||
id := r.PathValue("runId")
|
||||
if err := contract.ValidateID(id); err != nil {
|
||||
writeError(w, err)
|
||||
return
|
||||
}
|
||||
view, err := deps.Runs.GetRun(r.Context(), id)
|
||||
if err != nil {
|
||||
writeError(w, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, view)
|
||||
}))
|
||||
mux.HandleFunc(contract.GetFailuresRoute, method(http.MethodGet, func(w http.ResponseWriter, r *http.Request) {
|
||||
id := r.PathValue("runId")
|
||||
if err := contract.ValidateID(id); err != nil {
|
||||
writeError(w, err)
|
||||
return
|
||||
}
|
||||
page, err := parsePage(r.URL.RawQuery)
|
||||
if err != nil {
|
||||
writeError(w, err)
|
||||
return
|
||||
}
|
||||
result, err := deps.Failures.ListFailures(r.Context(), id, page)
|
||||
if err != nil {
|
||||
writeError(w, err)
|
||||
return
|
||||
}
|
||||
if result.Items == nil {
|
||||
result.Items = []contract.Failure{}
|
||||
}
|
||||
writeJSON(w, http.StatusOK, result)
|
||||
}))
|
||||
mux.HandleFunc(contract.PostReplayRoute, method(http.MethodPost, func(w http.ResponseWriter, r *http.Request) {
|
||||
if deps.Simple != nil {
|
||||
id := r.PathValue("runId")
|
||||
if err := contract.ValidateID(id); err != nil {
|
||||
writeError(w, err)
|
||||
return
|
||||
}
|
||||
var request struct {
|
||||
RequestedBy string `json:"requestedBy"`
|
||||
MaxRounds int `json:"maxRounds"`
|
||||
}
|
||||
decoder := json.NewDecoder(http.MaxBytesReader(w, r.Body, maxReplayBodyBytes))
|
||||
decoder.DisallowUnknownFields()
|
||||
if err := decoder.Decode(&request); err != nil {
|
||||
writeError(w, contract.ErrInvalidArgument)
|
||||
return
|
||||
}
|
||||
if err := decoder.Decode(new(any)); err != io.EOF {
|
||||
writeError(w, contract.ErrInvalidArgument)
|
||||
return
|
||||
}
|
||||
accepted, err := deps.Simple.Request(r.Context(), id, request.RequestedBy, request.MaxRounds)
|
||||
if errors.Is(err, fullsyncruntime.ErrNoFailures) {
|
||||
writeAPIError(w, http.StatusUnprocessableEntity, "no_failures", "Run has no failures to replay")
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
writeError(w, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusAccepted, accepted)
|
||||
return
|
||||
}
|
||||
request, err := parseReplay(w, r)
|
||||
if err != nil {
|
||||
var tooLarge *http.MaxBytesError
|
||||
if errors.As(err, &tooLarge) {
|
||||
writeAPIError(w, http.StatusRequestEntityTooLarge, "body_too_large", "Replay request body is too large")
|
||||
} else {
|
||||
writeError(w, err)
|
||||
}
|
||||
return
|
||||
}
|
||||
accepted, err := deps.Replays.RequestReplay(r.Context(), request)
|
||||
if err != nil {
|
||||
writeError(w, err)
|
||||
return
|
||||
}
|
||||
// Construct a local status URL from the returned identity, never an
|
||||
// externally supplied redirect. Invalid service output is a server error.
|
||||
accepted.StatusURL, err = contract.ActionStatusURL(accepted.ActionID)
|
||||
if err != nil || contract.ValidateID(accepted.TargetRunID) != nil || !accepted.Status.Valid() {
|
||||
writeAPIError(w, http.StatusInternalServerError, "internal_error", "Internal server error")
|
||||
return
|
||||
}
|
||||
w.Header().Set("Location", accepted.StatusURL)
|
||||
writeJSON(w, http.StatusAccepted, accepted)
|
||||
}))
|
||||
mux.HandleFunc("/internal/v1/full-sync-replays/{replayRunId}", method(http.MethodGet, func(w http.ResponseWriter, r *http.Request) {
|
||||
if deps.Simple == nil {
|
||||
writeError(w, contract.ErrNotFound)
|
||||
return
|
||||
}
|
||||
status, err := deps.Simple.Status(r.Context(), r.PathValue("replayRunId"))
|
||||
if err != nil {
|
||||
writeError(w, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, status)
|
||||
}))
|
||||
mux.HandleFunc(contract.GetActionRoute, method(http.MethodGet, func(w http.ResponseWriter, r *http.Request) {
|
||||
if deps.Simple != nil {
|
||||
writeError(w, contract.ErrNotFound)
|
||||
return
|
||||
}
|
||||
id := r.PathValue("actionId")
|
||||
if err := contract.ValidateID(id); err != nil {
|
||||
writeError(w, err)
|
||||
return
|
||||
}
|
||||
action, err := deps.Actions.GetReplayAction(r.Context(), id)
|
||||
if err != nil {
|
||||
writeError(w, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, action)
|
||||
}))
|
||||
mux.HandleFunc("/", func(w http.ResponseWriter, _ *http.Request) {
|
||||
writeError(w, contract.ErrNotFound)
|
||||
})
|
||||
if bearerToken == "" {
|
||||
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Cache-Control", "no-store")
|
||||
mux.ServeHTTP(w, r)
|
||||
}), nil
|
||||
}
|
||||
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Cache-Control", "no-store")
|
||||
values := r.Header.Values("Authorization")
|
||||
if len(values) == 1 {
|
||||
parts := strings.Fields(values[0])
|
||||
if len(parts) == 2 && strings.EqualFold(parts[0], "Bearer") && subtle.ConstantTimeCompare([]byte(parts[1]), []byte(bearerToken)) == 1 {
|
||||
mux.ServeHTTP(w, r)
|
||||
return
|
||||
}
|
||||
}
|
||||
w.Header().Set("WWW-Authenticate", `Bearer realm="full-sync"`)
|
||||
writeAPIError(w, http.StatusUnauthorized, "unauthorized", "Valid bearer authentication is required")
|
||||
}), nil
|
||||
}
|
||||
|
||||
func method(expected string, next http.HandlerFunc) http.HandlerFunc {
|
||||
return func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.Method != expected {
|
||||
w.Header().Set("Allow", expected)
|
||||
writeAPIError(w, http.StatusMethodNotAllowed, "method_not_allowed", "Method not allowed")
|
||||
return
|
||||
}
|
||||
next(w, r)
|
||||
}
|
||||
}
|
||||
|
||||
func parsePage(rawQuery string) (contract.PageRequest, error) {
|
||||
query, err := url.ParseQuery(rawQuery)
|
||||
if err != nil || len(query["cursor"]) > 1 || len(query["limit"]) > 1 {
|
||||
return contract.PageRequest{}, gerror.Wrap(contract.ErrInvalidArgument, "invalid pagination query")
|
||||
}
|
||||
page := contract.PageRequest{Cursor: contract.Cursor(query.Get("cursor"))}
|
||||
if value := query.Get("limit"); value != "" {
|
||||
page.Limit, err = strconv.Atoi(value)
|
||||
if err != nil {
|
||||
return contract.PageRequest{}, gerror.Wrap(contract.ErrInvalidArgument, "limit must be an integer")
|
||||
}
|
||||
}
|
||||
return page.Normalized()
|
||||
}
|
||||
|
||||
func parseReplay(w http.ResponseWriter, r *http.Request) (contract.ReplayRequest, error) {
|
||||
var request contract.ReplayRequest
|
||||
if value := r.Header.Get("Content-Type"); value != "" {
|
||||
mediaType, _, err := mime.ParseMediaType(value)
|
||||
if err != nil || mediaType != "application/json" {
|
||||
return request, gerror.Wrap(contract.ErrInvalidArgument, "replay body must be JSON")
|
||||
}
|
||||
}
|
||||
data, err := io.ReadAll(http.MaxBytesReader(w, r.Body, maxReplayBodyBytes))
|
||||
if err != nil {
|
||||
var tooLarge *http.MaxBytesError
|
||||
if errors.As(err, &tooLarge) {
|
||||
return request, err
|
||||
}
|
||||
return request, gerror.Wrap(contract.ErrInvalidArgument, "cannot read replay body")
|
||||
}
|
||||
if !utf8.Valid(data) {
|
||||
return request, gerror.Wrap(contract.ErrInvalidArgument, "replay body must be UTF-8")
|
||||
}
|
||||
decoder := json.NewDecoder(bytes.NewReader(data))
|
||||
decoder.DisallowUnknownFields()
|
||||
if err := decoder.Decode(&request); err != nil {
|
||||
return request, gerror.Wrap(contract.ErrInvalidArgument, "invalid replay JSON")
|
||||
}
|
||||
if err := decoder.Decode(new(any)); err != io.EOF {
|
||||
return request, gerror.Wrap(contract.ErrInvalidArgument, "replay body must contain exactly one JSON object")
|
||||
}
|
||||
if len(r.Header.Values(contract.IdempotencyHeader)) != 1 {
|
||||
return request, gerror.Wrap(contract.ErrInvalidArgument, "exactly one Idempotency-Key is required")
|
||||
}
|
||||
request.SourceRunID = r.PathValue("runId")
|
||||
request.IdempotencyKey = r.Header.Get(contract.IdempotencyHeader)
|
||||
return request, request.Validate()
|
||||
}
|
||||
|
||||
func writeError(w http.ResponseWriter, err error) {
|
||||
switch {
|
||||
case errors.Is(err, contract.ErrInvalidArgument):
|
||||
writeAPIError(w, http.StatusBadRequest, "invalid_argument", "Invalid request")
|
||||
case errors.Is(err, contract.ErrNotFound):
|
||||
writeAPIError(w, http.StatusNotFound, "not_found", "Resource not found")
|
||||
case errors.Is(err, contract.ErrFailureDetailsUnavailable):
|
||||
writeAPIError(w, http.StatusGone, "failure_details_unavailable", "Live failure details are no longer available")
|
||||
case errors.Is(err, contract.ErrConflict), errors.Is(err, contract.ErrLeaseLost):
|
||||
writeAPIError(w, http.StatusConflict, "conflict", "Request conflicts with the current resource")
|
||||
case errors.Is(err, contract.ErrBusy):
|
||||
writeAPIError(w, http.StatusServiceUnavailable, "busy", "Resource is busy; retry later")
|
||||
default:
|
||||
writeAPIError(w, http.StatusInternalServerError, "internal_error", "Internal server error")
|
||||
}
|
||||
}
|
||||
|
||||
func writeAPIError(w http.ResponseWriter, status int, code, message string) {
|
||||
writeJSON(w, status, ErrorResponse{Error: APIError{Code: code, Message: message}})
|
||||
}
|
||||
|
||||
func writeJSON(w http.ResponseWriter, status int, value any) {
|
||||
// Marshal before writing headers so an invalid service payload cannot
|
||||
// produce a successful status with truncated JSON.
|
||||
data, err := json.Marshal(value)
|
||||
if err != nil {
|
||||
status = http.StatusInternalServerError
|
||||
data = []byte(`{"error":{"code":"internal_error","message":"Internal server error"}}`)
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json; charset=utf-8")
|
||||
w.WriteHeader(status)
|
||||
_, _ = w.Write(append(data, '\n'))
|
||||
}
|
||||
@@ -0,0 +1,437 @@
|
||||
package httpapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"git.5iot.com/iotlcm-public/iotlcm-worker-svr/internal/fullsync/contract"
|
||||
)
|
||||
|
||||
const (
|
||||
testToken = "test-secret"
|
||||
runPath = "/internal/v1/full-sync-runs/run-1"
|
||||
failurePath = runPath + "/failures"
|
||||
replayPath = runPath + "/replay"
|
||||
actionPath = "/internal/v1/full-sync-actions/action-1"
|
||||
)
|
||||
|
||||
type runFunc func(context.Context, string) (contract.RunView, error)
|
||||
|
||||
func (f runFunc) GetRun(ctx context.Context, id string) (contract.RunView, error) {
|
||||
return f(ctx, id)
|
||||
}
|
||||
|
||||
type runListFunc func(context.Context, contract.RunListFilter) (contract.RunList, error)
|
||||
|
||||
func (f runListFunc) ListRuns(ctx context.Context, filter contract.RunListFilter) (contract.RunList, error) {
|
||||
return f(ctx, filter)
|
||||
}
|
||||
|
||||
type failureFunc func(context.Context, string, contract.PageRequest) (contract.FailurePage, error)
|
||||
|
||||
func (f failureFunc) ListFailures(ctx context.Context, id string, page contract.PageRequest) (contract.FailurePage, error) {
|
||||
return f(ctx, id, page)
|
||||
}
|
||||
|
||||
type replayFunc func(context.Context, contract.ReplayRequest) (contract.ReplayAccepted, error)
|
||||
|
||||
func (f replayFunc) RequestReplay(ctx context.Context, req contract.ReplayRequest) (contract.ReplayAccepted, error) {
|
||||
return f(ctx, req)
|
||||
}
|
||||
|
||||
type actionFunc func(context.Context, string) (contract.ReplayAction, error)
|
||||
|
||||
func (f actionFunc) GetReplayAction(ctx context.Context, id string) (contract.ReplayAction, error) {
|
||||
return f(ctx, id)
|
||||
}
|
||||
|
||||
func testDependencies() Dependencies {
|
||||
return Dependencies{
|
||||
Runs: runFunc(func(context.Context, string) (contract.RunView, error) {
|
||||
return contract.RunView{}, contract.ErrNotFound
|
||||
}),
|
||||
List: runListFunc(func(context.Context, contract.RunListFilter) (contract.RunList, error) {
|
||||
return contract.RunList{}, contract.ErrNotFound
|
||||
}),
|
||||
Failures: failureFunc(func(context.Context, string, contract.PageRequest) (contract.FailurePage, error) {
|
||||
return contract.FailurePage{}, contract.ErrNotFound
|
||||
}),
|
||||
Replays: replayFunc(func(context.Context, contract.ReplayRequest) (contract.ReplayAccepted, error) {
|
||||
return contract.ReplayAccepted{}, contract.ErrNotFound
|
||||
}),
|
||||
Actions: actionFunc(func(context.Context, string) (contract.ReplayAction, error) {
|
||||
return contract.ReplayAction{}, contract.ErrNotFound
|
||||
}),
|
||||
}
|
||||
}
|
||||
|
||||
func testHandler(t *testing.T, deps Dependencies) http.Handler {
|
||||
t.Helper()
|
||||
handler, err := NewHandler(testToken, deps)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return handler
|
||||
}
|
||||
|
||||
func request(handler http.Handler, method, path, body string, headers ...string) *httptest.ResponseRecorder {
|
||||
req := httptest.NewRequest(method, path, strings.NewReader(body))
|
||||
req.Header.Set("Authorization", "Bearer "+testToken)
|
||||
for i := 0; i < len(headers); i += 2 {
|
||||
req.Header.Set(headers[i], headers[i+1])
|
||||
}
|
||||
recorder := httptest.NewRecorder()
|
||||
handler.ServeHTTP(recorder, req)
|
||||
return recorder
|
||||
}
|
||||
|
||||
func assertError(t *testing.T, response *httptest.ResponseRecorder, status int, code string) {
|
||||
t.Helper()
|
||||
if response.Code != status {
|
||||
t.Fatalf("status = %d, want %d: %s", response.Code, status, response.Body)
|
||||
}
|
||||
if !strings.HasPrefix(response.Header().Get("Content-Type"), "application/json") {
|
||||
t.Fatalf("missing JSON content type: %v", response.Header())
|
||||
}
|
||||
var body ErrorResponse
|
||||
if err := json.Unmarshal(response.Body.Bytes(), &body); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if body.Error.Code != code || body.Error.Message == "" {
|
||||
t.Fatalf("error = %+v, want code %q and message", body, code)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAuthenticationProtectsEveryRoute(t *testing.T) {
|
||||
deps := testDependencies()
|
||||
deps.Runs = runFunc(func(context.Context, string) (contract.RunView, error) {
|
||||
t.Fatal("unauthenticated request reached service")
|
||||
return contract.RunView{}, nil
|
||||
})
|
||||
handler := testHandler(t, deps)
|
||||
for _, path := range []string{runPath, failurePath, replayPath, actionPath, "/unknown"} {
|
||||
for _, auth := range []string{"", "Bearer", "Bearer wrong", "Basic " + testToken, "Bearer " + testToken + " extra"} {
|
||||
t.Run(path+"/"+auth, func(t *testing.T) {
|
||||
response := request(handler, http.MethodGet, path, "", "Authorization", auth)
|
||||
assertError(t, response, 401, "unauthorized")
|
||||
if response.Header().Get("WWW-Authenticate") == "" {
|
||||
t.Fatal("missing authentication challenge")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
req := httptest.NewRequest(http.MethodGet, runPath, nil)
|
||||
req.Header.Add("Authorization", "Bearer "+testToken)
|
||||
req.Header.Add("Authorization", "Bearer "+testToken)
|
||||
response := httptest.NewRecorder()
|
||||
handler.ServeHTTP(response, req)
|
||||
assertError(t, response, 401, "unauthorized")
|
||||
}
|
||||
|
||||
func TestRunListQuery(t *testing.T) {
|
||||
deps := testDependencies()
|
||||
deps.List = runListFunc(func(_ context.Context, filter contract.RunListFilter) (contract.RunList, error) {
|
||||
if filter.SyncKind != contract.SyncKindTrafficUsage || filter.Limit != 7 {
|
||||
t.Fatalf("filter = %+v", filter)
|
||||
}
|
||||
return contract.RunList{Items: []contract.RunView{{Source: contract.SourceLive, Run: contract.RunMeta{RunID: "run-1", Operation: "current_month_usage_full_sync"}}}, Total: 1}, nil
|
||||
})
|
||||
response := request(testHandler(t, deps), http.MethodGet, "/internal/v1/full-sync-runs?syncKind=traffic_usage&limit=7", "")
|
||||
if response.Code != http.StatusOK || !strings.Contains(response.Body.String(), "run-1") {
|
||||
t.Fatalf("response = %d %s", response.Code, response.Body)
|
||||
}
|
||||
for _, query := range []string{"syncKind=invalid", "limit=101", "limit=abc", "limit=1&limit=2"} {
|
||||
assertError(t, request(testHandler(t, deps), http.MethodGet, "/internal/v1/full-sync-runs?"+query, ""), 400, "invalid_argument")
|
||||
}
|
||||
}
|
||||
|
||||
func TestRoutesAndValidation(t *testing.T) {
|
||||
handler := testHandler(t, testDependencies())
|
||||
for _, route := range []struct{ method, path string }{
|
||||
{http.MethodGet, runPath}, {http.MethodGet, failurePath},
|
||||
{http.MethodPost, replayPath}, {http.MethodGet, actionPath},
|
||||
} {
|
||||
t.Run(route.path, func(t *testing.T) {
|
||||
response := request(handler, route.method, route.path, `{"requestedBy":"operator"}`, contract.IdempotencyHeader, "key")
|
||||
assertError(t, response, 404, "not_found")
|
||||
wrongMethod := http.MethodPost
|
||||
if route.method == http.MethodPost {
|
||||
wrongMethod = http.MethodGet
|
||||
}
|
||||
response = request(handler, wrongMethod, route.path, "")
|
||||
assertError(t, response, 405, "method_not_allowed")
|
||||
if response.Header().Get("Allow") != route.method {
|
||||
t.Fatalf("Allow = %q", response.Header().Get("Allow"))
|
||||
}
|
||||
})
|
||||
}
|
||||
for _, path := range []string{"/unknown", "/internal/v1/full-sync-runs", runPath + "/other", runPath + "/"} {
|
||||
assertError(t, request(handler, http.MethodGet, path, ""), 404, "not_found")
|
||||
}
|
||||
for _, path := range []string{
|
||||
"/internal/v1/full-sync-runs/%20run",
|
||||
"/internal/v1/full-sync-runs/a%2Fb/failures",
|
||||
"/internal/v1/full-sync-actions/%7Baction%7D",
|
||||
} {
|
||||
assertError(t, request(handler, http.MethodGet, path, ""), 400, "invalid_argument")
|
||||
}
|
||||
assertError(t, request(handler, http.MethodPost, "/internal/v1/full-sync-runs/a%5Cb/replay", `{"requestedBy":"operator"}`, contract.IdempotencyHeader, "key"), 400, "invalid_argument")
|
||||
}
|
||||
|
||||
func TestRunAndActionResponsesPropagateContext(t *testing.T) {
|
||||
type contextKey struct{}
|
||||
deps := testDependencies()
|
||||
deps.Runs = runFunc(func(ctx context.Context, id string) (contract.RunView, error) {
|
||||
if id != "run-1" || ctx.Value(contextKey{}) != "request" {
|
||||
t.Fatal("run identity or request context lost")
|
||||
}
|
||||
return contract.RunView{Source: contract.SourceLive, Run: contract.RunMeta{RunID: id}}, nil
|
||||
})
|
||||
deps.Actions = actionFunc(func(ctx context.Context, id string) (contract.ReplayAction, error) {
|
||||
if id != "action-1" || ctx.Value(contextKey{}) != "request" {
|
||||
t.Fatal("action identity or request context lost")
|
||||
}
|
||||
return contract.ReplayAction{ActionID: id, Status: contract.ActionProcessing, IdempotencyKey: "private-key", Fingerprint: "private-fingerprint", LeaseToken: "private-lease", Revision: 99}, nil
|
||||
})
|
||||
handler := testHandler(t, deps)
|
||||
for _, path := range []string{runPath, actionPath} {
|
||||
req := httptest.NewRequest(http.MethodGet, path, nil).WithContext(context.WithValue(context.Background(), contextKey{}, "request"))
|
||||
req.Header.Set("Authorization", "bearer "+testToken)
|
||||
response := httptest.NewRecorder()
|
||||
handler.ServeHTTP(response, req)
|
||||
if response.Code != 200 || response.Header().Get("Cache-Control") != "no-store" {
|
||||
t.Fatalf("response = %d %s", response.Code, response.Body)
|
||||
}
|
||||
if strings.Contains(response.Body.String(), "private-") || strings.Contains(response.Body.String(), "revision") {
|
||||
t.Fatalf("internal fields leaked: %s", response.Body)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestFailurePagination(t *testing.T) {
|
||||
for _, test := range []struct {
|
||||
query string
|
||||
limit int
|
||||
cursor contract.Cursor
|
||||
}{
|
||||
{"", 100, ""}, {"?limit=", 100, ""}, {"?limit=0", 100, ""},
|
||||
{"?limit=1", 1, ""}, {"?limit=500", 500, ""},
|
||||
{"?limit=17&cursor=" + url.QueryEscape("opaque/+ token==:雪"), 17, "opaque/+ token==:雪"},
|
||||
} {
|
||||
t.Run(test.query, func(t *testing.T) {
|
||||
deps := testDependencies()
|
||||
calls := 0
|
||||
deps.Failures = failureFunc(func(_ context.Context, id string, page contract.PageRequest) (contract.FailurePage, error) {
|
||||
calls++
|
||||
if id != "run-1" || page.Limit != test.limit || page.Cursor != test.cursor {
|
||||
t.Fatalf("query = %q %+v", id, page)
|
||||
}
|
||||
return contract.FailurePage{NextCursor: "next:opaque/+="}, nil
|
||||
})
|
||||
response := request(testHandler(t, deps), http.MethodGet, failurePath+test.query, "")
|
||||
if response.Code != 200 || calls != 1 {
|
||||
t.Fatalf("response = %d, calls = %d", response.Code, calls)
|
||||
}
|
||||
var page contract.FailurePage
|
||||
if err := json.Unmarshal(response.Body.Bytes(), &page); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if page.Items == nil || len(page.Items) != 0 || page.NextCursor != "next:opaque/+=" {
|
||||
t.Fatalf("empty scan page lost array or continuation: %s", response.Body)
|
||||
}
|
||||
})
|
||||
}
|
||||
deps := testDependencies()
|
||||
deps.Failures = failureFunc(func(context.Context, string, contract.PageRequest) (contract.FailurePage, error) {
|
||||
t.Fatal("invalid pagination reached service")
|
||||
return contract.FailurePage{}, nil
|
||||
})
|
||||
handler := testHandler(t, deps)
|
||||
for _, query := range []string{"limit=-1", "limit=501", "limit=abc", "limit=1.5", "limit=999999999999999999999999", "limit=1&limit=2", "cursor=a&cursor=b", "cursor=%ZZ", "cursor=a;b"} {
|
||||
assertError(t, request(handler, http.MethodGet, failurePath+"?"+query, ""), 400, "invalid_argument")
|
||||
}
|
||||
}
|
||||
|
||||
func TestFailureCursorValidationAndLiveOnly(t *testing.T) {
|
||||
deps := testDependencies()
|
||||
deps.Runs = runFunc(func(context.Context, string) (contract.RunView, error) {
|
||||
t.Fatal("failure route must not query run summaries or fall back to archives")
|
||||
return contract.RunView{}, nil
|
||||
})
|
||||
deps.Failures = failureFunc(func(_ context.Context, _ string, page contract.PageRequest) (contract.FailurePage, error) {
|
||||
if page.Cursor == "another-run-token" {
|
||||
return contract.FailurePage{}, fmt.Errorf("wrong query binding: %w", contract.ErrInvalidArgument)
|
||||
}
|
||||
return contract.FailurePage{}, fmt.Errorf("expired details: %w", contract.ErrFailureDetailsUnavailable)
|
||||
})
|
||||
handler := testHandler(t, deps)
|
||||
assertError(t, request(handler, http.MethodGet, failurePath+"?cursor=another-run-token", ""), 400, "invalid_argument")
|
||||
assertError(t, request(handler, http.MethodGet, failurePath, ""), 410, "failure_details_unavailable")
|
||||
}
|
||||
|
||||
func TestReplayValidation(t *testing.T) {
|
||||
deps := testDependencies()
|
||||
deps.Replays = replayFunc(func(context.Context, contract.ReplayRequest) (contract.ReplayAccepted, error) {
|
||||
t.Fatal("invalid replay reached service")
|
||||
return contract.ReplayAccepted{}, nil
|
||||
})
|
||||
handler := testHandler(t, deps)
|
||||
for _, body := range []string{"", "{", "null", "[]", `{}`, `{"requestedBy":""}`, `{"requestedBy":" "}`, `{"requestedBy":12}`, `{"requestedBy":"a","targetRunId":"injected"}`, `{"requestedBy":"a"} {}`, `{"requestedBy":"a"} trailing`, "{\"requestedBy\":\"\xff\"}"} {
|
||||
t.Run(body, func(t *testing.T) {
|
||||
assertError(t, request(handler, http.MethodPost, replayPath, body, contract.IdempotencyHeader, "key"), 400, "invalid_argument")
|
||||
})
|
||||
}
|
||||
for _, key := range []string{"", " ", strings.Repeat("x", contract.MaxIdempotencyKeyBytes+1), "bad\tkey"} {
|
||||
assertError(t, request(handler, http.MethodPost, replayPath, `{"requestedBy":"operator"}`, contract.IdempotencyHeader, key), 400, "invalid_argument")
|
||||
}
|
||||
assertError(t, request(handler, http.MethodPost, replayPath, `{"requestedBy":"operator"}`), 400, "invalid_argument")
|
||||
assertError(t, request(handler, http.MethodPost, replayPath, `{"requestedBy":"operator"}`, contract.IdempotencyHeader, "key", "Content-Type", "text/plain"), 400, "invalid_argument")
|
||||
assertError(t, request(handler, http.MethodPost, replayPath, `{"requestedBy":"`+strings.Repeat("x", maxReplayBodyBytes)+`"}`, contract.IdempotencyHeader, "key"), 413, "body_too_large")
|
||||
req := httptest.NewRequest(http.MethodPost, replayPath, strings.NewReader(`{"requestedBy":"operator"}`))
|
||||
req.Header.Set("Authorization", "Bearer "+testToken)
|
||||
req.Header.Add(contract.IdempotencyHeader, "key")
|
||||
req.Header.Add(contract.IdempotencyHeader, "key")
|
||||
response := httptest.NewRecorder()
|
||||
handler.ServeHTTP(response, req)
|
||||
assertError(t, response, 400, "invalid_argument")
|
||||
}
|
||||
|
||||
// The mock owns idempotency; the HTTP adapter must pass exact request identity
|
||||
// and return the service's existing action/target even across handler instances.
|
||||
func TestReplayAcceptedIdempotencyAndConflict(t *testing.T) {
|
||||
deps := testDependencies()
|
||||
var (
|
||||
stored contract.ReplayRequest
|
||||
calls int
|
||||
)
|
||||
deps.Replays = replayFunc(func(_ context.Context, req contract.ReplayRequest) (contract.ReplayAccepted, error) {
|
||||
calls++
|
||||
if calls == 1 {
|
||||
stored = req
|
||||
}
|
||||
if req != stored {
|
||||
return contract.ReplayAccepted{}, fmt.Errorf("changed fingerprint: %w", contract.ErrConflict)
|
||||
}
|
||||
return contract.ReplayAccepted{ActionID: "action-1", TargetRunID: "replay-1", Status: contract.ActionAccepted, StatusURL: "https://untrusted.invalid"}, nil
|
||||
})
|
||||
var first contract.ReplayAccepted
|
||||
for i := 0; i < 2; i++ {
|
||||
response := request(testHandler(t, deps), http.MethodPost, replayPath, "{\"requestedBy\":\" operator \"}\n", contract.IdempotencyHeader, "key", "Content-Type", "application/json; charset=utf-8")
|
||||
if response.Code != 202 {
|
||||
t.Fatalf("response = %d %s", response.Code, response.Body)
|
||||
}
|
||||
var accepted contract.ReplayAccepted
|
||||
if err := json.Unmarshal(response.Body.Bytes(), &accepted); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if accepted.ActionID != "action-1" || accepted.TargetRunID != "replay-1" || accepted.StatusURL != actionPath || response.Header().Get("Location") != actionPath {
|
||||
t.Fatalf("acceptance = %+v, headers = %v", accepted, response.Header())
|
||||
}
|
||||
if i == 0 {
|
||||
first = accepted
|
||||
} else if !reflect.DeepEqual(first, accepted) {
|
||||
t.Fatalf("repeat changed identity: %+v vs %+v", first, accepted)
|
||||
}
|
||||
}
|
||||
if stored.SourceRunID != "run-1" || stored.RequestedBy != " operator " || stored.IdempotencyKey != "key" {
|
||||
t.Fatalf("request identity changed: %+v", stored)
|
||||
}
|
||||
assertError(t, request(testHandler(t, deps), http.MethodPost, replayPath, `{"requestedBy":"operator"}`, contract.IdempotencyHeader, "key"), 409, "conflict")
|
||||
if calls != 3 {
|
||||
t.Fatalf("replay calls = %d", calls)
|
||||
}
|
||||
}
|
||||
|
||||
func TestWrappedServiceErrorsUseConsistentEnvelope(t *testing.T) {
|
||||
for _, test := range []struct {
|
||||
err error
|
||||
status int
|
||||
code string
|
||||
}{
|
||||
{contract.ErrInvalidArgument, 400, "invalid_argument"},
|
||||
{contract.ErrNotFound, 404, "not_found"},
|
||||
{contract.ErrConflict, 409, "conflict"},
|
||||
{contract.ErrLeaseLost, 409, "conflict"},
|
||||
{contract.ErrFailureDetailsUnavailable, 410, "failure_details_unavailable"},
|
||||
{contract.ErrBusy, 503, "busy"},
|
||||
{fmt.Errorf("private database password"), 500, "internal_error"},
|
||||
} {
|
||||
t.Run(test.code, func(t *testing.T) {
|
||||
err := fmt.Errorf("private database detail: %w", test.err)
|
||||
deps := Dependencies{
|
||||
Runs: runFunc(func(context.Context, string) (contract.RunView, error) { return contract.RunView{}, err }),
|
||||
List: runListFunc(func(context.Context, contract.RunListFilter) (contract.RunList, error) {
|
||||
return contract.RunList{}, err
|
||||
}),
|
||||
Failures: failureFunc(func(context.Context, string, contract.PageRequest) (contract.FailurePage, error) {
|
||||
return contract.FailurePage{}, err
|
||||
}),
|
||||
Replays: replayFunc(func(context.Context, contract.ReplayRequest) (contract.ReplayAccepted, error) {
|
||||
return contract.ReplayAccepted{}, err
|
||||
}),
|
||||
Actions: actionFunc(func(context.Context, string) (contract.ReplayAction, error) { return contract.ReplayAction{}, err }),
|
||||
}
|
||||
handler := testHandler(t, deps)
|
||||
for _, path := range []string{runPath, failurePath, actionPath, replayPath} {
|
||||
method := http.MethodGet
|
||||
if path == replayPath {
|
||||
method = http.MethodPost
|
||||
}
|
||||
response := request(handler, method, path, `{"requestedBy":"operator"}`, contract.IdempotencyHeader, "key")
|
||||
assertError(t, response, test.status, test.code)
|
||||
if strings.Contains(response.Body.String(), "private") {
|
||||
t.Fatal("service details leaked")
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestInvalidServiceOutputReturns500(t *testing.T) {
|
||||
deps := testDependencies()
|
||||
deps.Replays = replayFunc(func(context.Context, contract.ReplayRequest) (contract.ReplayAccepted, error) {
|
||||
return contract.ReplayAccepted{ActionID: "bad/id"}, nil
|
||||
})
|
||||
deps.Runs = runFunc(func(context.Context, string) (contract.RunView, error) {
|
||||
return contract.RunView{Rounds: []contract.RoundStats{{ErrorCounts: contract.ErrorCounts{contract.ErrorOther: -1}}}}, nil
|
||||
})
|
||||
handler := testHandler(t, deps)
|
||||
assertError(t, request(handler, http.MethodPost, replayPath, `{"requestedBy":"operator"}`, contract.IdempotencyHeader, "key"), 500, "internal_error")
|
||||
assertError(t, request(handler, http.MethodGet, runPath, ""), 500, "internal_error")
|
||||
}
|
||||
|
||||
func TestNewHandlerRejectsInvalidConfiguration(t *testing.T) {
|
||||
if _, err := NewHandler("", testDependencies()); err != nil {
|
||||
t.Fatalf("empty token should use Docker-network-only access: %v", err)
|
||||
}
|
||||
for _, token := range []string{" ", "a b", "token\n"} {
|
||||
if _, err := NewHandler(token, testDependencies()); err == nil {
|
||||
t.Errorf("accepted token %q", token)
|
||||
}
|
||||
}
|
||||
for i := 0; i < 5; i++ {
|
||||
deps := testDependencies()
|
||||
switch i {
|
||||
case 0:
|
||||
deps.Runs = nil
|
||||
case 1:
|
||||
deps.Failures = nil
|
||||
case 2:
|
||||
deps.Replays = nil
|
||||
case 3:
|
||||
deps.Actions = nil
|
||||
case 4:
|
||||
deps.List = nil
|
||||
}
|
||||
if _, err := NewHandler(testToken, deps); err == nil {
|
||||
t.Errorf("accepted missing dependency %d", i)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,48 @@
|
||||
package httpapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
|
||||
"git.5iot.com/iotlcm-public/iotlcm-worker-svr/internal/fullsync/contract"
|
||||
|
||||
"github.com/gogf/gf/v2/errors/gerror"
|
||||
)
|
||||
|
||||
type runReader struct {
|
||||
live contract.LiveRunReader
|
||||
archive contract.ArchivedRunReader
|
||||
rounds contract.RoundStatsReader
|
||||
}
|
||||
|
||||
// NewRunReader composes live-first run queries without knowing either storage
|
||||
// implementation. Archive and rounds are optional (nil disables that lookup).
|
||||
// Only absent live metadata permits archive fallback; live round errors do not.
|
||||
func NewRunReader(live contract.LiveRunReader, archive contract.ArchivedRunReader, rounds contract.RoundStatsReader) (contract.RunReader, error) {
|
||||
if live == nil {
|
||||
return nil, gerror.New("full-sync run reader requires a live reader")
|
||||
}
|
||||
return &runReader{live: live, archive: archive, rounds: rounds}, nil
|
||||
}
|
||||
|
||||
func (r *runReader) GetRun(ctx context.Context, runID string) (contract.RunView, error) {
|
||||
if err := contract.ValidateID(runID); err != nil {
|
||||
return contract.RunView{}, err
|
||||
}
|
||||
meta, err := r.live.GetRunMeta(ctx, runID)
|
||||
if err == nil {
|
||||
view := contract.RunView{Source: contract.SourceLive, Run: meta}
|
||||
if r.rounds != nil {
|
||||
view.Rounds, err = r.rounds.GetRoundStats(ctx, runID)
|
||||
}
|
||||
return view, err
|
||||
}
|
||||
if !errors.Is(err, contract.ErrNotFound) || r.archive == nil {
|
||||
return contract.RunView{}, err
|
||||
}
|
||||
archive, err := r.archive.GetArchivedRun(ctx, runID)
|
||||
if err != nil {
|
||||
return contract.RunView{}, err
|
||||
}
|
||||
return contract.RunView{Source: contract.SourceArchive, Run: archive.Run, Rounds: archive.Rounds}, nil
|
||||
}
|
||||
@@ -0,0 +1,125 @@
|
||||
package httpapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"testing"
|
||||
|
||||
"git.5iot.com/iotlcm-public/iotlcm-worker-svr/internal/fullsync/contract"
|
||||
)
|
||||
|
||||
type liveFunc func(context.Context, string) (contract.RunMeta, error)
|
||||
|
||||
func (f liveFunc) GetRunMeta(ctx context.Context, id string) (contract.RunMeta, error) {
|
||||
return f(ctx, id)
|
||||
}
|
||||
|
||||
type archiveFunc func(context.Context, string) (contract.RunArchive, error)
|
||||
|
||||
func (f archiveFunc) GetArchivedRun(ctx context.Context, id string) (contract.RunArchive, error) {
|
||||
return f(ctx, id)
|
||||
}
|
||||
|
||||
type roundsFunc func(context.Context, string) ([]contract.RoundStats, error)
|
||||
|
||||
func (f roundsFunc) GetRoundStats(ctx context.Context, id string) ([]contract.RoundStats, error) {
|
||||
return f(ctx, id)
|
||||
}
|
||||
|
||||
func TestRunLookupLiveFirstAndArchiveFallback(t *testing.T) {
|
||||
for _, test := range []struct {
|
||||
name string
|
||||
liveErr error
|
||||
roundsErr error
|
||||
archiveErr error
|
||||
wantStatus int
|
||||
wantSource contract.RunSource
|
||||
wantArchives int
|
||||
wantRounds int
|
||||
}{
|
||||
{name: "live", wantStatus: 200, wantSource: contract.SourceLive, wantRounds: 1},
|
||||
{name: "archive", liveErr: fmt.Errorf("expired: %w", contract.ErrNotFound), wantStatus: 200, wantSource: contract.SourceArchive, wantArchives: 1},
|
||||
{name: "unknown", liveErr: contract.ErrNotFound, archiveErr: contract.ErrNotFound, wantStatus: 404, wantArchives: 1},
|
||||
{name: "live failure never falls back", liveErr: fmt.Errorf("live offline"), wantStatus: 500},
|
||||
{name: "round error never falls back", roundsErr: fmt.Errorf("rounds offline"), wantStatus: 500, wantRounds: 1},
|
||||
{name: "missing rounds never fall back", roundsErr: contract.ErrNotFound, wantStatus: 404, wantRounds: 1},
|
||||
{name: "archive error", liveErr: contract.ErrNotFound, archiveErr: fmt.Errorf("archive offline"), wantStatus: 500, wantArchives: 1},
|
||||
} {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
var liveCalls, archiveCalls, roundsCalls int
|
||||
reader, err := NewRunReader(
|
||||
liveFunc(func(_ context.Context, id string) (contract.RunMeta, error) {
|
||||
liveCalls++
|
||||
if id != "run-1" {
|
||||
t.Fatalf("live ID = %q", id)
|
||||
}
|
||||
return contract.RunMeta{RunID: id, CurrentRound: 2}, test.liveErr
|
||||
}),
|
||||
archiveFunc(func(_ context.Context, id string) (contract.RunArchive, error) {
|
||||
archiveCalls++
|
||||
if id != "run-1" {
|
||||
t.Fatalf("archive ID = %q", id)
|
||||
}
|
||||
return contract.RunArchive{Run: contract.RunMeta{RunID: id, CurrentRound: 3}, Rounds: []contract.RoundStats{{RunID: id, Round: 3}}}, test.archiveErr
|
||||
}),
|
||||
roundsFunc(func(_ context.Context, id string) ([]contract.RoundStats, error) {
|
||||
roundsCalls++
|
||||
if id != "run-1" {
|
||||
t.Fatalf("rounds ID = %q", id)
|
||||
}
|
||||
return []contract.RoundStats{{RunID: id, Round: 2}}, test.roundsErr
|
||||
}),
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
deps := testDependencies()
|
||||
deps.Runs = reader
|
||||
response := request(testHandler(t, deps), http.MethodGet, runPath, "")
|
||||
if response.Code != test.wantStatus {
|
||||
t.Fatalf("response = %d %s", response.Code, response.Body)
|
||||
}
|
||||
if liveCalls != 1 || archiveCalls != test.wantArchives || roundsCalls != test.wantRounds {
|
||||
t.Fatalf("calls: live=%d archive=%d rounds=%d", liveCalls, archiveCalls, roundsCalls)
|
||||
}
|
||||
if response.Code == 200 {
|
||||
var view contract.RunView
|
||||
if err := json.Unmarshal(response.Body.Bytes(), &view); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if view.Source != test.wantSource || view.Run.RunID != "run-1" || len(view.Rounds) != 1 || view.Rounds[0].Round != view.Run.CurrentRound {
|
||||
t.Fatalf("view = %+v", view)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunReaderOptionalServicesAndValidation(t *testing.T) {
|
||||
if _, err := NewRunReader(nil, nil, nil); err == nil {
|
||||
t.Fatal("accepted nil live reader")
|
||||
}
|
||||
calls := 0
|
||||
reader, err := NewRunReader(liveFunc(func(_ context.Context, id string) (contract.RunMeta, error) {
|
||||
calls++
|
||||
if id == "missing" {
|
||||
return contract.RunMeta{}, contract.ErrNotFound
|
||||
}
|
||||
return contract.RunMeta{RunID: id}, nil
|
||||
}), nil, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := reader.GetRun(context.Background(), "bad/id"); err == nil || calls != 0 {
|
||||
t.Fatal("invalid ID reached live reader")
|
||||
}
|
||||
view, err := reader.GetRun(context.Background(), "run-1")
|
||||
if err != nil || view.Source != contract.SourceLive || view.Rounds != nil {
|
||||
t.Fatalf("view = %+v, err = %v", view, err)
|
||||
}
|
||||
if _, err := reader.GetRun(context.Background(), "missing"); err != contract.ErrNotFound {
|
||||
t.Fatalf("error = %v", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,119 @@
|
||||
package httpapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"net"
|
||||
"net/http"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/gogf/gf/v2/errors/gerror"
|
||||
)
|
||||
|
||||
// Config is deliberately independent of worker/config-file wiring. Addr must
|
||||
// be explicit; use 127.0.0.1:0 to let the OS allocate a local test port.
|
||||
type Config struct {
|
||||
Addr string
|
||||
BearerToken string // Optional for standalone callers; production Worker leaves it empty.
|
||||
ShutdownTimeout time.Duration
|
||||
}
|
||||
|
||||
// Server is single-use. Start reports bind errors synchronously; after a
|
||||
// successful Start, Wait reports serve or shutdown errors. Canceling the Start
|
||||
// context stops accepting requests and drains active requests up to the timeout.
|
||||
type Server struct {
|
||||
server *http.Server
|
||||
shutdownTimeout time.Duration
|
||||
mu sync.Mutex
|
||||
started bool
|
||||
addr net.Addr
|
||||
done chan struct{}
|
||||
err error
|
||||
}
|
||||
|
||||
func NewServer(config Config, deps Dependencies) (*Server, error) {
|
||||
if config.Addr == "" || config.ShutdownTimeout < 0 {
|
||||
return nil, gerror.New("full-sync HTTP requires an address and a nonnegative shutdown timeout")
|
||||
}
|
||||
handler, err := NewHandler(config.BearerToken, deps)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if config.ShutdownTimeout == 0 {
|
||||
config.ShutdownTimeout = 10 * time.Second
|
||||
}
|
||||
return &Server{
|
||||
server: &http.Server{
|
||||
Addr: config.Addr, Handler: handler,
|
||||
ReadHeaderTimeout: 5 * time.Second,
|
||||
ReadTimeout: 30 * time.Second,
|
||||
WriteTimeout: 30 * time.Second,
|
||||
IdleTimeout: 60 * time.Second,
|
||||
},
|
||||
shutdownTimeout: config.ShutdownTimeout,
|
||||
done: make(chan struct{}),
|
||||
}, nil
|
||||
}
|
||||
|
||||
func (s *Server) Start(ctx context.Context) error {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
if s.started {
|
||||
return gerror.New("full-sync HTTP server has already been started")
|
||||
}
|
||||
s.started = true
|
||||
if err := ctx.Err(); err != nil {
|
||||
s.err = err
|
||||
close(s.done)
|
||||
return err
|
||||
}
|
||||
listener, err := (&net.ListenConfig{}).Listen(ctx, "tcp", s.server.Addr)
|
||||
if err != nil {
|
||||
s.err = gerror.Wrap(err, "listen for full-sync HTTP")
|
||||
close(s.done)
|
||||
return s.err
|
||||
}
|
||||
s.addr = listener.Addr()
|
||||
go s.run(ctx, listener)
|
||||
return nil
|
||||
}
|
||||
|
||||
// Addr returns the bound address, or nil before a successful Start.
|
||||
func (s *Server) Addr() net.Addr {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
return s.addr
|
||||
}
|
||||
|
||||
// Wait may be called concurrently after Start (including a failed Start).
|
||||
func (s *Server) Wait() error {
|
||||
<-s.done
|
||||
return s.err
|
||||
}
|
||||
|
||||
func (s *Server) run(ctx context.Context, listener net.Listener) {
|
||||
serveDone := make(chan error, 1)
|
||||
go func() { serveDone <- s.server.Serve(listener) }()
|
||||
var err error
|
||||
select {
|
||||
case err = <-serveDone:
|
||||
// An unexpected listener failure also needs to drain active requests.
|
||||
case <-ctx.Done():
|
||||
}
|
||||
shutdownCtx, cancel := context.WithTimeout(context.Background(), s.shutdownTimeout)
|
||||
shutdownErr := s.server.Shutdown(shutdownCtx)
|
||||
cancel()
|
||||
if shutdownErr != nil {
|
||||
// Shutdown does not close active connections when its deadline expires.
|
||||
_ = s.server.Close()
|
||||
}
|
||||
if err == nil {
|
||||
err = <-serveDone
|
||||
}
|
||||
if errors.Is(err, http.ErrServerClosed) {
|
||||
err = nil
|
||||
}
|
||||
s.err = errors.Join(err, shutdownErr)
|
||||
close(s.done)
|
||||
}
|
||||
@@ -0,0 +1,226 @@
|
||||
package httpapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"net"
|
||||
"net/http"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"git.5iot.com/iotlcm-public/iotlcm-worker-svr/internal/fullsync/contract"
|
||||
)
|
||||
|
||||
func awaitServer(t *testing.T, done <-chan error) error {
|
||||
t.Helper()
|
||||
select {
|
||||
case err := <-done:
|
||||
return err
|
||||
case <-time.After(5 * time.Second):
|
||||
t.Fatal("server did not stop")
|
||||
return nil
|
||||
}
|
||||
}
|
||||
|
||||
func startTestServer(t *testing.T, deps Dependencies, timeout time.Duration) (*Server, context.CancelFunc, <-chan error) {
|
||||
t.Helper()
|
||||
server, err := NewServer(Config{Addr: "127.0.0.1:0", BearerToken: testToken, ShutdownTimeout: timeout}, deps)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if server.Addr() != nil {
|
||||
t.Fatal("address before Start")
|
||||
}
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
t.Cleanup(cancel)
|
||||
if err := server.Start(ctx); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
done := make(chan error, 1)
|
||||
go func() { done <- server.Wait() }()
|
||||
return server, cancel, done
|
||||
}
|
||||
|
||||
func serverRequest(server *Server) (*http.Response, error) {
|
||||
req, err := http.NewRequest(http.MethodGet, "http://"+server.Addr().String()+runPath, nil)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
req.Header.Set("Authorization", "Bearer "+testToken)
|
||||
client := &http.Client{Timeout: 4 * time.Second}
|
||||
return client.Do(req)
|
||||
}
|
||||
|
||||
func TestServerStartServeAndCancel(t *testing.T) {
|
||||
server, cancel, done := startTestServer(t, testDependencies(), 0)
|
||||
response, err := serverRequest(server)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
_ = response.Body.Close()
|
||||
if response.StatusCode != 404 {
|
||||
t.Fatalf("status = %d", response.StatusCode)
|
||||
}
|
||||
if err := server.Start(context.Background()); err == nil {
|
||||
t.Fatal("duplicate Start succeeded")
|
||||
}
|
||||
cancel()
|
||||
if err := awaitServer(t, done); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := server.Wait(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
connection, err := net.DialTimeout("tcp", server.Addr().String(), time.Second)
|
||||
if err == nil {
|
||||
_ = connection.Close()
|
||||
t.Fatal("listener still accepts after shutdown")
|
||||
}
|
||||
}
|
||||
|
||||
func TestServerGracefulShutdownDrainsActiveRequest(t *testing.T) {
|
||||
var (
|
||||
entered = make(chan struct{})
|
||||
release = make(chan struct{})
|
||||
once sync.Once
|
||||
)
|
||||
t.Cleanup(func() { once.Do(func() { close(release) }) })
|
||||
deps := testDependencies()
|
||||
deps.Runs = runFunc(func(ctx context.Context, id string) (contract.RunView, error) {
|
||||
close(entered)
|
||||
select {
|
||||
case <-release:
|
||||
return contract.RunView{Source: contract.SourceLive, Run: contract.RunMeta{RunID: id}}, nil
|
||||
case <-ctx.Done():
|
||||
return contract.RunView{}, ctx.Err()
|
||||
}
|
||||
})
|
||||
server, cancel, done := startTestServer(t, deps, 2*time.Second)
|
||||
requestDone := make(chan error, 1)
|
||||
go func() {
|
||||
response, err := serverRequest(server)
|
||||
if err == nil {
|
||||
_, err = io.ReadAll(response.Body)
|
||||
_ = response.Body.Close()
|
||||
if response.StatusCode != 200 {
|
||||
err = errors.New("active request did not complete successfully")
|
||||
}
|
||||
}
|
||||
requestDone <- err
|
||||
}()
|
||||
select {
|
||||
case <-entered:
|
||||
case <-time.After(3 * time.Second):
|
||||
t.Fatal("request did not enter handler")
|
||||
}
|
||||
cancel()
|
||||
select {
|
||||
case err := <-done:
|
||||
t.Fatalf("shutdown returned before draining: %v", err)
|
||||
case <-time.After(50 * time.Millisecond):
|
||||
}
|
||||
once.Do(func() { close(release) })
|
||||
if err := awaitServer(t, requestDone); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := awaitServer(t, done); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestServerShutdownDeadlineClosesActiveConnections(t *testing.T) {
|
||||
var (
|
||||
entered = make(chan struct{})
|
||||
exited = make(chan struct{})
|
||||
)
|
||||
deps := testDependencies()
|
||||
deps.Runs = runFunc(func(ctx context.Context, _ string) (contract.RunView, error) {
|
||||
close(entered)
|
||||
<-ctx.Done()
|
||||
close(exited)
|
||||
return contract.RunView{}, ctx.Err()
|
||||
})
|
||||
server, cancel, done := startTestServer(t, deps, 50*time.Millisecond)
|
||||
requestDone := make(chan error, 1)
|
||||
go func() {
|
||||
response, err := serverRequest(server)
|
||||
if response != nil {
|
||||
_ = response.Body.Close()
|
||||
}
|
||||
requestDone <- err
|
||||
}()
|
||||
select {
|
||||
case <-entered:
|
||||
case <-time.After(3 * time.Second):
|
||||
t.Fatal("request did not enter handler")
|
||||
}
|
||||
cancel()
|
||||
if err := awaitServer(t, done); !errors.Is(err, context.DeadlineExceeded) {
|
||||
t.Fatalf("shutdown error = %v", err)
|
||||
}
|
||||
select {
|
||||
case <-exited:
|
||||
case <-time.After(3 * time.Second):
|
||||
t.Fatal("forced close did not cancel request context")
|
||||
}
|
||||
_ = awaitServer(t, requestDone)
|
||||
}
|
||||
|
||||
func TestServerStartupFailureIsSynchronous(t *testing.T) {
|
||||
listener, err := net.Listen("tcp", "127.0.0.1:0")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(func() { _ = listener.Close() })
|
||||
server, err := NewServer(Config{Addr: listener.Addr().String(), BearerToken: testToken}, testDependencies())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
startErr := server.Start(context.Background())
|
||||
if startErr == nil {
|
||||
t.Fatal("Start did not report occupied port")
|
||||
}
|
||||
if server.Wait() != startErr || server.Addr() != nil {
|
||||
t.Fatal("failed startup state was lost")
|
||||
}
|
||||
var netErr *net.OpError
|
||||
if !errors.As(startErr, &netErr) {
|
||||
t.Fatalf("network error cause lost: %v", startErr)
|
||||
}
|
||||
}
|
||||
|
||||
func TestServerCanceledBeforeStartAndImmediateCancellation(t *testing.T) {
|
||||
server, err := NewServer(Config{Addr: "127.0.0.1:0", BearerToken: testToken}, testDependencies())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
cancel()
|
||||
if err := server.Start(ctx); !errors.Is(err, context.Canceled) {
|
||||
t.Fatalf("Start = %v", err)
|
||||
}
|
||||
if err := server.Wait(); !errors.Is(err, context.Canceled) {
|
||||
t.Fatalf("Wait = %v", err)
|
||||
}
|
||||
// Cancellation can win the race before Serve starts accepting requests.
|
||||
for i := 0; i < 10; i++ {
|
||||
_, cancel, done := startTestServer(t, testDependencies(), time.Second)
|
||||
cancel()
|
||||
if err := awaitServer(t, done); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestNewServerRejectsInvalidConfiguration(t *testing.T) {
|
||||
for _, config := range []Config{
|
||||
{BearerToken: testToken},
|
||||
{Addr: "127.0.0.1:0", BearerToken: testToken, ShutdownTimeout: -time.Second},
|
||||
} {
|
||||
if _, err := NewServer(config, testDependencies()); err == nil {
|
||||
t.Fatalf("accepted config %+v", config)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,59 @@
|
||||
package httpapi
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"git.5iot.com/iotlcm-public/iotlcm-worker-svr/internal/fullsync/runtime"
|
||||
"git.5iot.com/iotlcm-public/iotlcm-worker-svr/internal/queue/zset"
|
||||
"git.5iot.com/iotlcm-public/iotlcm-worker-svr/internal/testutil"
|
||||
)
|
||||
|
||||
func TestSimpleReplayHTTPPrivateNetworkAcceptance(t *testing.T) {
|
||||
redis := testutil.Redis(t)
|
||||
store, _ := zset.New(redis, "sim-status:")
|
||||
service, _ := runtime.NewSimpleReplay(store)
|
||||
ctx := context.Background()
|
||||
key, _ := store.RunMetaKey("source")
|
||||
failed, _ := store.FailedKey("source")
|
||||
_, _ = redis.Do(ctx, "HSET", key, "runId", "source", "status", "completed_with_failures", "producerDone", "1", "currentRound", "1")
|
||||
record := zset.FailureRecord{TaskID: "task", CardID: 1, ICCID: "card", ChanKey: "ch", TempKey: "template"}
|
||||
value, _ := json.Marshal(record)
|
||||
_, _ = redis.Do(ctx, "HSET", failed, "task", string(value))
|
||||
deps := testDependencies()
|
||||
deps.Simple = service
|
||||
handler, err := NewHandler("", deps)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
req := httptest.NewRequest(http.MethodPost, "/internal/v1/full-sync-runs/source/replay", bytes.NewBufferString(`{"requestedBy":"42"}`))
|
||||
w := httptest.NewRecorder()
|
||||
handler.ServeHTTP(w, req)
|
||||
if w.Code != 202 {
|
||||
t.Fatalf("accepted: %d %s", w.Code, w.Body.String())
|
||||
}
|
||||
var receipt runtime.ReplayReceipt
|
||||
if err := json.Unmarshal(w.Body.Bytes(), &receipt); err != nil || receipt.ReplayRunID == "" {
|
||||
t.Fatalf("receipt: %+v %v", receipt, err)
|
||||
}
|
||||
deadline := time.Now().Add(3 * time.Second)
|
||||
for time.Now().Before(deadline) {
|
||||
w := httptest.NewRecorder()
|
||||
req := httptest.NewRequest(http.MethodGet, "/internal/v1/full-sync-replays/"+receipt.ReplayRunID, nil)
|
||||
handler.ServeHTTP(w, req)
|
||||
if w.Code == 200 {
|
||||
var status runtime.ReplayStatus
|
||||
_ = json.Unmarshal(w.Body.Bytes(), &status)
|
||||
if status.Status == "completed" && status.Enqueued == 1 {
|
||||
return
|
||||
}
|
||||
}
|
||||
time.Sleep(20 * time.Millisecond)
|
||||
}
|
||||
t.Fatal("replay HTTP never completed")
|
||||
}
|
||||
Reference in New Issue
Block a user