Commit Graph

74 Commits

Author SHA1 Message Date
douxu 2fbef3e1fa feat: add manual measurement synchronization workflow
- add configurable CL3611 and IEC104 manual sync clients
- synchronize measurement mode and value updates with consistent timestamps
- move Redis change sets into the repository layer
- nest manual sync configuration under dataRT
- improve startup transaction error handling and JSONB type mapping
- update Kubernetes and MongoDB deployment documentation
2026-08-04 17:12:17 +08:00
douxu de1110905e feat: bootstrap measurement data objects in Redis
- load measurement hierarchy and metadata from PostgreSQL
- generate 3 types Redis hashes
- map measurement type enums to TM, TS, TC, TA, and SP
- validate token uniqueness, mode, type, and JSONB fields
- track initialized hashes and clean up stale measurement keys
- initialize measurement data objects during modelRT startup
2026-07-29 15:12:05 +08:00
douxu 8679bfe82a feat: bootstrap parameter data objects from PostgreSQL to Redis
- resolve component and dynamic parameter attributes through project mappings
- generate full and local parameter token hashes during startup
- validate parameter routes, descriptions, duplicate records, and token ambiguity
- track initialized Redis keys for stale cache cleanup
- centralize supported parameter attribute groups
- distinguish CL3611 phasor and sampled-data identifiers
- add parameter initialization and SQL mapping tests
2026-07-28 16:24:29 +08:00
douxu 72e2143cb3 feat: merge hotfix/measurement-recommend-fuzzy-offset into develop 2026-07-14 10:12:51 +08:00
douxu b05219ef4c refactor: replace topology tree with graph cache
- replace globalTree startup loading with globalTopologyGraph
- add topologyGraph adjacency cache with in/out edges, start nodes, and end nodes
- remove multiBranchTreeNode and legacy tree-building helpers
- make topology queries independent of the all-zero UUID virtual root
- keep TOPOLOGY_ANALYSIS on per-task DB querying while clarifying point-to-point reachability results
- add coverage for multi-parent topology reachability
2026-07-07 10:01:14 +08:00
douxu 491c10e8c5 feat: add component column recommendations for config tokens
- add component as a token6 config recommendation constant
- read component table column names during startup
- bind component config recommendations to component table columns
- add full and local autocomplete terms for component column paths
2026-07-01 14:37:13 +08:00
douxu 2650771cdb docs: regenerate swagger and drop stale TODO comment
- regenerate AsyncTask swagger responses, keeping only the documented case
  - remove obsolete 200/400/404/500 response entries from docs/swagger
  - delete commented-out termsOfService TODO from main.go API annotations
2026-06-23 11:27:42 +08:00
douxu c6545e29ba style: normalize log messages to lowercase across task package
- lowercase first letter of all logger.Info/Warn/Error message strings
    in task/worker.go, task/retry_queue.go, task/handler_factory.go,
    task/metrics_logger.go, task/retry_manager.go, task/queue_producer.go,
    task/initializer.go, task/test_task.go, and main.go
  - fix inline comments in main.go that mixed Chinese and uppercase English
  - align Dockerfile comment casing with project convention
2026-06-01 15:50:11 +08:00
douxu 4a2666aa3b fix: correct caller frames in GORM logger and DB arg in main
- add *Skip variants (logSkip, makeLogFieldsSkip, getLoggerCallerInfoSkip)
    so wrapper functions report the true call site, not logger internals
  - switch GormLogger.Trace to use ErrorSkip/WarnSkip/InfoSkip with extraSkip=1
    so SQL log lines point to GORM caller rather than the logger facade
  - pass postgresDBClient instead of tx to GetFullMeasurementSet in main
2026-05-19 17:38:22 +08:00
douxu d051c161b7 perf: parallelize GetFullMeasurementSet with errgroup
- run 5 independent DB queries concurrently via errgroup.WithContext
  - add ctx parameter and bind db with WithContext for cancellation support
  - replace silent error swallowing (if err == nil) with wrapped error returns
  - promote golang.org/x/sync to direct dependency in go.mod
2026-05-18 16:49:46 +08:00
douxu 42956d1793 feat: add dedicated message-exchange for task lifecycle notifications
- add constants/message.go with MessageTask* categories and message-exchange /
    message-queue / dead-letter routing constants
  - add mq/publish_message.go with PushMessageToRabbitMQ (confirm mode,
    dead-letter queue) separate from the existing event-exchange publisher
  - add mq/emit.go with TryEmitMessage for non-blocking, OTel-traced dispatch
  - add mq/event/task_event_gen.go with NewTaskSubmitted/Running/Completed/
    Failed/CancelledMessage constructors
  - wire TryEmitMessage into task worker and create/cancel handlers so all 5
    lifecycle transitions are published (previously task.* routed to
    event-exchange with no matching binding, causing silent drops)
  - harden Dockerfile: scratch final image, pinned alpine:3.21 certs stage,
    apk upgrade in builder, add -trimpath -mod=readonly go build flags
  - add full K8s manifests under deploy/k8s/ for Redis, RabbitMQ (mTLS),
    ModelRT (Downward API, scratch image, readOnlyRootFilesystem), Jaeger,
    Loki, Promtail, Grafana
  - expand deploy.md with async_task SQL schema, TLS cert generation steps,
    K8s deployment procedures, and SSH tunnel configuration
2026-05-13 16:58:36 +08:00
douxu cccd4becdc feat: add Loki logging, fix MQ shutdown order, improve realtime tracing
- add LokiConfig and batching lokiSyncer for dev-mode direct log push
  - refactor zap logger to support mode-aware encoding and K8s pod fields
  - fix RabbitMQ shutdown race: move CloseRabbitProxy to defer so channel
    closes before connection (prevents 504 error on Ctrl+C)
  - wrap MsgChan with EventMessage to carry per-cycle trace carrier
  - create new root OTel span per computation cycle linked to startup span,
    giving each cycle an independent traceID with startup as reference
2026-05-11 17:34:27 +08:00
douxu 1b1f43db7f feat: implement topology analysis async task with BFS connectivity check
- add TopologyAnalysisHandler.Execute() with 5-phase BFS reachability
    check between start/end component UUIDs; support CheckInService flag
    to skip out-of-service nodes during traversal
  - carry task params through RabbitMQ message (TaskQueueMessage.Params)
    instead of re-querying DB in handler; update TaskHandler.Execute
    interface and all handler signatures accordingly
  - fix BuildMultiBranchTree UUIDFrom condition bug; return nodeMap for
    O(1) lookup; add QueryTopologicByStartUUID for directed traversal
  - add QueryBayByUUID/QueryBaysByUUIDs and
    QueryComponentsInServiceByUUIDs (two-column select) to database layer
  - add diagram.FindPath via LCA algorithm for tree path reconstruction
  - move initTracerProvider to middleware.InitTracerProvider; add
    OtelConfig struct to ModelRTConfig for endpoint configuration
  - update topology analysis params to start/end_component_uuid +
    check_in_service; remove dead topology init code
2026-04-24 17:14:46 +08:00
douxu 03bd058558 feat: implement end-to-end distributed tracing for HTTP and async tasks
- introduce typed traceCtxKey to prevent context key collisions (staticcheck fix)
  - inject B3 trace values into c.Request.Context() in StartTrace middleware
    so handlers using c.Request.Context() carry trace info
  - create startup trace context in main.go, replacing context.TODO()
  - propagate HTTP traceID/spanID through TaskQueueMessage into RabbitMQ
    worker, linking HTTP request → publish → execution on the same traceID
  - fix GORM logger null traceID by binding ctx to AutoMigrate and queries
    via db.WithContext(ctx)
  - thread ctx through handler factory to fix null traceID in startup logs
  - replace per-request RabbitMQ producer with channel-based
    PushTaskToRabbitMQ goroutine; restrict Swagger to non-production
2026-04-23 16:48:32 +08:00
douxu f8c0951a13 Extend async task system with database integration and retry management
- Add AsyncTaskConfig to config structure
  - Create database operations for task state management (async_task_extended.go)
  - Add configuration middleware for Gin context
  - Extract task worker initialization to separate file (initializer.go)
  - Implement retry strategies with exponential backoff (retry_manager.go)
  - Add retry queue for failed task scheduling (retry_queue.go)
  - Enhance worker metrics with detailed per-task-type tracking
  - Integrate database operations into task worker for status updates
  - Add comprehensive metrics logging system
2026-04-03 10:07:43 +08:00
douxu 9e4c35794c implemented task queue publishing using RabbitMQ
Added configuration middleware integration
  Added retry logic for queue publishing
  Added task worker initialization (main.go):

Created initTaskWorker function for task worker configuration
  Added worker startup and shutdown logic
  Added CORS middleware configuration
  Registered config middleware
2026-04-01 17:15:33 +08:00
douxu 4b52e5f3c6 optimize code of event record and push rabbitmq func 2026-02-28 17:38:33 +08:00
douxu f6bb3fb985 optimize code of push event to rabbitmq 2026-02-26 16:48:12 +08:00
douxu 2ececc38d9 optimzie code organization structure of rabbitmq event 2026-02-25 17:14:25 +08:00
douxu 6618209bcc optimzie code of rabbitmq connection 2026-02-06 17:45:59 +08:00
douxu 9be984899c optimize code of push event alarm func 2026-02-03 17:05:32 +08:00
douxu 2126aa7b06 optimize code of config 2026-01-29 17:00:20 +08:00
douxu 3374eec047 optimize code of redis init 2026-01-28 16:49:12 +08:00
douxu fcf4ef3f7d fix bug of token4-token7 type attribute recommend api 2025-12-31 16:52:40 +08:00
douxu 36e196bedd add nspath filter of recommend api 2025-12-30 16:35:29 +08:00
douxu 7969861746 optimize func of cleanup recommend redis cache 2025-12-26 17:10:22 +08:00
douxu 8e4bdfd0e9 add fullpath completion of component attribute group recommend 2025-12-26 12:00:00 +08:00
douxu 42751c1020 optimize the logic for loading the cache of measurement nodes for traversing components 2025-12-25 17:17:20 +08:00
douxu 41e2998739 optimize deploy doc and optimize TraverseAttributeGroupTables func 2025-12-22 17:38:15 +08:00
douxu 9499e579b3 optimize code of sql struct and measurement node recommend api 2025-12-19 17:33:12 +08:00
douxu 689d31c246 optimize dockerfile and config generate of docker deploy 2025-12-17 17:09:20 +08:00
douxu 4f5d998659 optimzie rate limit of measurement group api 2025-12-17 14:12:35 +08:00
douxu 716f56babb optimize docker deploy file 2025-12-12 11:01:18 +08:00
douxu 5021e7fda1 comment out unused kafka code 2025-12-12 10:23:04 +08:00
douxu befb4e8971 fix bug of server deploy 2025-12-11 16:42:25 +08:00
douxu 3f70be0d1c fix bug of new version of measurement recommend api 2025-12-06 18:32:00 +08:00
douxu a7d894d2de write code for real time data compute shell 2025-11-28 17:17:58 +08:00
douxu d434a7737d optimize variable naming and init real time data compute func 2025-11-13 17:29:49 +08:00
douxu 041d7e5788 optimize variable naming and optimize real time data computing api 2025-11-12 17:34:18 +08:00
douxu 54128bedac fix bug of measurement recommend api 2025-10-20 15:06:23 +08:00
douxu 86199269f8 add deploy.md of deploy modelRT project 2025-10-17 17:10:10 +08:00
douxu 62e897190d optimize code of measurement recommend and logger output 2025-10-15 17:08:32 +08:00
douxu bcf80842b0 fix bug of main.go 2025-10-14 16:12:00 +08:00
douxu 0c09e7bd25 add func of generate service token 2025-09-12 17:12:02 +08:00
douxu 55a606a3f3 add redis zset structure 2025-09-09 16:02:36 +08:00
douxu f4ab4e4ea4 refactor(orm/circuit_diagram_component): fix compilation issues caused by structure field changes
http://server.baseware.net:9000/project/datart/task/47
2025-08-15 16:25:48 +08:00
douxu 65e0c5da92 optimize modelRT routing structure 2025-07-31 10:31:26 +08:00
douxu a70f77464c refactor(gorm-logger): 1. add gorm logger in gorm config 2.use faced func in gorm logger 2025-06-23 16:00:48 +08:00
douxu 9aa5b0dcc6 refactor(logger): 1. optimize the logger log module design and add link tracking related designs
2. add logger facade functions to simplify the use of alarm functions
2025-06-05 15:56:40 +08:00
douxu d2196701ec fix(multi-branch-tree-of-topologic): add global tree variable and fix topologic info processing bug 2025-05-20 16:08:17 +08:00