Commit Graph

235 Commits

Author SHA1 Message Date
douxu c4e892f1c7 fix: correct typo in Jaeger K8s service name
- rename `jaeger-serivce` to `jaeger-service` in jaeger-service.yaml
2026-06-08 17:05:21 +08:00
douxu 195150d9b1 fix: fix K8s service names, deployment command, and GORM logger
- rename all K8s services to xxx-service convention and update
    all configmap references (postgres, mongodb, loki, jaeger)
  - add explicit command: ["/app/modelrt"] to deployment to prevent
    args from being treated as the executable (no ENTRYPOINT in
    Dockerfile)
  - set deploy_env to development to bypass Redis empty-password
    guard in non-production Minikube environment
  - fix GormLogger Info/Warn/Error to use fmt.Sprintf(msg, data...)
    so GORM printf-style messages are formatted correctly and avoid
    json: unsupported type: func() time.Time serialization panic
  - expand pg PVC storage from 2Gi to 6Gi
  - rename loop variable msg to task in PushTaskToRabbitMQ for clarity
  - align comment indentation in queue_producer.go
2026-06-03 17:11:54 +08:00
douxu 3309e53653 docs: document Dockerfile smoke tests and load workflow for Minikube
- add 3-stage build table (builder/certs/scratch) with image size note
  - add build-arg USER_ID override example in section 5.1
  - add section 5.1.1 with smoke-test commands (size check, inspect, dry
    run, full start)
  - add workflow for loading pre-built local images into Minikube
    directly
  - bump builder base image from golang:1.25-alpine to
    golang:1.26-alpine
  - normalize inline Dockerfile comments to lowercase
  - remove example config COPY from final scratch stage
2026-06-02 16:35:13 +08:00
douxu c6545e29ba style: normalize log messages to lowercase across task package
- lowercase first letter of all logger.Info/Warn/Error message strings
    in task/worker.go, task/retry_queue.go, task/handler_factory.go,
    task/metrics_logger.go, task/retry_manager.go, task/queue_producer.go,
    task/initializer.go, task/test_task.go, and main.go
  - fix inline comments in main.go that mixed Chinese and uppercase English
  - align Dockerfile comment casing with project convention
2026-06-01 15:50:11 +08:00
douxu c17ddb80b9 feat: merge bay-realtime-data-calc into develop 2026-06-01 14:10:10 +08:00
douxu 57d1111a83 refactor: modernize Go idioms and add MongoDB K8s manifests
- replace interface{} with any across ~30 files for Go 1.18+ style
  - adopt for-range-over-int loops in place of explicit index loops
  - use maps.Copy from stdlib to replace manual map copy loops
  - use min() builtin for exponential backoff delay cap in retry_manager
  - add MongoDB 7.0 K8s manifests (StatefulSet, Service, PVC, Secret)
  - document PostgreSQL and MongoDB deploy steps in deploy.md with SSH tunnel port mappings
2026-05-29 14:28:58 +08:00
douxu bacd43617e chore: bind sensitive config to env vars and bump Go image to 1.25
- bind postgres.password to POSTGRES_PASSWORD env var via viper BindEnv
  - bind service.secret_key to SERVICE_SECRET_KEY env var via viper BindEnv
  - upgrade builder base image from golang:1.24-alpine to golang:1.25-alpine
2026-05-29 10:56:17 +08:00
douxu 9c4dcd29e4 chore: bump Go to 1.26.3
- upgrade go directive in go.mod from 1.25.0 to 1.26.3
2026-05-28 16:36:51 +08:00
douxu 57371fbf1f docs: add Minikube PostgreSQL manifests and clean deploy markdown
- add split PostgreSQL K8s manifests for ConfigMap, Service, PVC, and StatefulSet
- expose PostgreSQL through NodePort for local Minikube access
- replace deploy.md LaTeX text syntax with Markdown inline code formatting
- keep deployment documentation rendering stable in Wiki.js and Markdown viewers
2026-05-27 16:51:00 +08:00
douxu 4a2666aa3b fix: correct caller frames in GORM logger and DB arg in main
- add *Skip variants (logSkip, makeLogFieldsSkip, getLoggerCallerInfoSkip)
    so wrapper functions report the true call site, not logger internals
  - switch GormLogger.Trace to use ErrorSkip/WarnSkip/InfoSkip with extraSkip=1
    so SQL log lines point to GORM caller rather than the logger facade
  - pass postgresDBClient instead of tx to GetFullMeasurementSet in main
2026-05-19 17:38:22 +08:00
douxu d051c161b7 perf: parallelize GetFullMeasurementSet with errgroup
- run 5 independent DB queries concurrently via errgroup.WithContext
  - add ctx parameter and bind db with WithContext for cancellation support
  - replace silent error swallowing (if err == nil) with wrapped error returns
  - promote golang.org/x/sync to direct dependency in go.mod
2026-05-18 16:49:46 +08:00
douxu 42956d1793 feat: add dedicated message-exchange for task lifecycle notifications
- add constants/message.go with MessageTask* categories and message-exchange /
    message-queue / dead-letter routing constants
  - add mq/publish_message.go with PushMessageToRabbitMQ (confirm mode,
    dead-letter queue) separate from the existing event-exchange publisher
  - add mq/emit.go with TryEmitMessage for non-blocking, OTel-traced dispatch
  - add mq/event/task_event_gen.go with NewTaskSubmitted/Running/Completed/
    Failed/CancelledMessage constructors
  - wire TryEmitMessage into task worker and create/cancel handlers so all 5
    lifecycle transitions are published (previously task.* routed to
    event-exchange with no matching binding, causing silent drops)
  - harden Dockerfile: scratch final image, pinned alpine:3.21 certs stage,
    apk upgrade in builder, add -trimpath -mod=readonly go build flags
  - add full K8s manifests under deploy/k8s/ for Redis, RabbitMQ (mTLS),
    ModelRT (Downward API, scratch image, readOnlyRootFilesystem), Jaeger,
    Loki, Promtail, Grafana
  - expand deploy.md with async_task SQL schema, TLS cert generation steps,
    K8s deployment procedures, and SSH tunnel configuration
2026-05-13 16:58:36 +08:00
douxu cccd4becdc feat: add Loki logging, fix MQ shutdown order, improve realtime tracing
- add LokiConfig and batching lokiSyncer for dev-mode direct log push
  - refactor zap logger to support mode-aware encoding and K8s pod fields
  - fix RabbitMQ shutdown race: move CloseRabbitProxy to defer so channel
    closes before connection (prevents 504 error on Ctrl+C)
  - wrap MsgChan with EventMessage to carry per-cycle trace carrier
  - create new root OTel span per computation cycle linked to startup span,
    giving each cycle an independent traceID with startup as reference
2026-05-11 17:34:27 +08:00
douxu 1dd8491440 refactor: replace EventStatusPersisted with IsPersisted field on EventRecord - add IsPersisted bool to EventRecord for explicit persistence tracking by eventRT consumer - remove EventStatusPersisted constant, decoupling DB persistence from event lifecycle status - update event status comments for accuracy and CIM-agnostic language 2026-05-08 16:19:12 +08:00
douxu 1ee722dd58 refactor: migrate trace propagation from B3 to W3C TraceContext - switch OTel propagator from b3.New() to propagation.TraceContext{} - rename B3 header constants to generic internal context keys
- remove go.opentelemetry.io/contrib/propagators/b3 dependency                                                                - add amqpHeaderCarrier to inject W3C traceparent into AMQP message headers
2026-05-07 16:43:34 +08:00
douxu 9661278935 refactor: rename TaskParams to Params and remove debug prints
- rename TaskParams interface to Params in task/base_task.go for brevity                                                     - remove debug fmt.Println/Printf statements from graph.go and handler_factory.go
  - fix is_local flag from false to true for existing test components in deploy.md                                             - add 6 new test component records (ns4-ns8) to deploy seed data
2026-04-28 17:41:28 +08:00
douxu 33f7d758e5 refactor: overhaul async task handler routing and fix data consistency
- fix params lost in RabbitMQ transit by threading them through PublishTask/PublishTaskWithRetry
  - fix UpdateTaskErrorInfo not setting status=FAILED on async_task
  - fix UpdateAsyncTaskResultWithError silently skipping when no result row exists (UPDATE → upsert)
  - sync task failure to async_task_result in updateTaskWithError
  - remove taskType from AsyncTaskHandler.Execute interface; rename TaskHandler → AsyncTaskHandler
  - replace CompositeHandler with direct factory.GetHandler dispatch via worker.dispatch()
  - use constructors (NewXxxHandler) for handler registration instead of zero-value literals
  - consolidate TaskType/TaskStatus/UnifiedTaskType into task/types.go; delete types_v2.go
  - extract BaseTask/TaskParams into task/base_task.go
2026-04-27 17:55:38 +08:00
douxu 1b1f43db7f feat: implement topology analysis async task with BFS connectivity check
- add TopologyAnalysisHandler.Execute() with 5-phase BFS reachability
    check between start/end component UUIDs; support CheckInService flag
    to skip out-of-service nodes during traversal
  - carry task params through RabbitMQ message (TaskQueueMessage.Params)
    instead of re-querying DB in handler; update TaskHandler.Execute
    interface and all handler signatures accordingly
  - fix BuildMultiBranchTree UUIDFrom condition bug; return nodeMap for
    O(1) lookup; add QueryTopologicByStartUUID for directed traversal
  - add QueryBayByUUID/QueryBaysByUUIDs and
    QueryComponentsInServiceByUUIDs (two-column select) to database layer
  - add diagram.FindPath via LCA algorithm for tree path reconstruction
  - move initTracerProvider to middleware.InitTracerProvider; add
    OtelConfig struct to ModelRTConfig for endpoint configuration
  - update topology analysis params to start/end_component_uuid +
    check_in_service; remove dead topology init code
2026-04-24 17:14:46 +08:00
douxu 03bd058558 feat: implement end-to-end distributed tracing for HTTP and async tasks
- introduce typed traceCtxKey to prevent context key collisions (staticcheck fix)
  - inject B3 trace values into c.Request.Context() in StartTrace middleware
    so handlers using c.Request.Context() carry trace info
  - create startup trace context in main.go, replacing context.TODO()
  - propagate HTTP traceID/spanID through TaskQueueMessage into RabbitMQ
    worker, linking HTTP request → publish → execution on the same traceID
  - fix GORM logger null traceID by binding ctx to AutoMigrate and queries
    via db.WithContext(ctx)
  - thread ctx through handler factory to fix null traceID in startup logs
  - replace per-request RabbitMQ producer with channel-based
    PushTaskToRabbitMQ goroutine; restrict Swagger to non-production
2026-04-23 16:48:32 +08:00
douxu 809e1cd87d Refactor: extract task constants to dedicated constants package
- Add constants/task.go with centralized task-related constants
    - Task priority levels (default, high, low)
    - Task queue configuration (exchange, queue, routing key)
    - Task message settings (max priority, TTL)
    - Task retry settings (max retries, delays)
    - Test task settings (sleep duration, max limit)

  - Update task-related files to use constants from constants package:
    - handler/async_task_create_handler.go
    - task/queue_message.go
    - task/queue_producer.go
    - task/retry_manager.go
    - task/test_task.go
    - task/types.go (add TypeTest)
    - task/worker.go
2026-04-22 17:20:26 +08:00
douxu 4a3f7a65bc Refactor async task handlers into specialized handlers
Split monolithic async_task_handler.go into separate handlers:
- async_task_cancel_handler.go: Handles task cancellation
- async_task_create_handler.go: Handles task creation
- async_task_progress_update_handler.go: Handles progress updates
- async_task_result_detail_handler.go: Handles result details
- async_task_result_query_handler.go: Handles result queries
- async_task_status_update_handler.go: Handles status updates

This improves code organization and maintainability by separating concerns.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-17 14:09:02 +08:00
douxu 4d5fcbc376 Refactor async task system with unified task interfaces and add test task type
- Create task/types_v2.go with unified task type definitions and interfaces
    * Add UnifiedTaskType and UnifiedTaskStatus constants
    * Define Task:Params interface for parameter validation and serialization
    * Define UnifiedTask interface as base for all task implementations
    * Add BaseTask for common task functionality
2026-04-14 17:00:30 +08:00
douxu f8c0951a13 Extend async task system with database integration and retry management
- Add AsyncTaskConfig to config structure
  - Create database operations for task state management (async_task_extended.go)
  - Add configuration middleware for Gin context
  - Extract task worker initialization to separate file (initializer.go)
  - Implement retry strategies with exponential backoff (retry_manager.go)
  - Add retry queue for failed task scheduling (retry_queue.go)
  - Enhance worker metrics with detailed per-task-type tracking
  - Integrate database operations into task worker for status updates
  - Add comprehensive metrics logging system
2026-04-03 10:07:43 +08:00
douxu 9e4c35794c implemented task queue publishing using RabbitMQ
Added configuration middleware integration
  Added retry logic for queue publishing
  Added task worker initialization (main.go):

Created initTaskWorker function for task worker configuration
  Added worker startup and shutdown logic
  Added CORS middleware configuration
  Registered config middleware
2026-04-01 17:15:33 +08:00
douxu 7ea66e48af add code of async task system 2026-03-20 15:00:04 +08:00
douxu de5f976c31 add route of async task system 2026-03-17 16:08:46 +08:00
douxu adcc8c6c91 add code of async task system 2026-03-13 11:45:22 +08:00
douxu 6e0d2186d8 optimize code of async task system 2026-03-12 16:37:06 +08:00
douxu a94abdb479 initialize the asynchronous task system's initial structure 2026-03-05 17:15:51 +08:00
douxu 898beaeec4 optimize struct of rabbitmq event 2026-03-02 17:00:09 +08:00
douxu 4b52e5f3c6 optimize code of event record and push rabbitmq func 2026-02-28 17:38:33 +08:00
douxu f6bb3fb985 optimize code of push event to rabbitmq 2026-02-26 16:48:12 +08:00
douxu 2ececc38d9 optimzie code organization structure of rabbitmq event 2026-02-25 17:14:25 +08:00
douxu 6c9da6fcd4 init event struct with option mode 2026-02-24 17:08:48 +08:00
douxu 56b9999d6b add constants varibale of power system events 2026-02-12 17:09:08 +08:00
douxu 1c385ee60d optimize code of rabbitmq connection and event alarm struct 2026-02-11 16:43:42 +08:00
douxu 6618209bcc optimzie code of rabbitmq connection 2026-02-06 17:45:59 +08:00
douxu 581153ed8d add git ignore item of mask certificate files 2026-02-05 17:01:16 +08:00
douxu f45b7d5fa4 optimize code of init rabbitmq connect func 2026-02-04 17:43:09 +08:00
douxu 9be984899c optimize code of push event alarm func 2026-02-03 17:05:32 +08:00
douxu 35cb969a54 add code of inter-module communication 2026-02-02 16:48:46 +08:00
douxu 02e0c9c31a optimzie of postgres db code 2026-01-30 17:42:50 +08:00
douxu 2126aa7b06 optimize code of config 2026-01-29 17:00:20 +08:00
douxu 3374eec047 optimize code of redis init 2026-01-28 16:49:12 +08:00
douxu 3ff29cc072 optimize code of real time data pull api 2026-01-28 14:03:25 +08:00
douxu 617d21500e optimize code of redis connenct func and real time data calculate 2026-01-27 17:41:17 +08:00
douxu 1a1727adab optimize reponse code and business code of measurement sub api 2026-01-26 16:29:50 +08:00
douxu fd2b202037 optimize code of websocket close handler 2026-01-22 16:19:00 +08:00
douxu 13433f93e5 optimize code of custom error structure 2026-01-20 16:42:54 +08:00
douxu e1886bc347 optimize code of judge wether success or failed return content 2026-01-19 16:39:35 +08:00