Skip to main content

Coginiti Architecture

Understanding Coginiti's architecture helps you make better decisions about how to structure your projects, optimize performance, and leverage the platform's capabilities effectively.

High-Level Architecture​

Coginiti is built as a modern, cloud-native DataOps platform with a containerized microservices architecture. The platform runs on Kubernetes and is designed for scalability, security, and performance across enterprise environments.

Container-Based Service Architecture​

Coginiti Team and Enterprise are deployed as containerized services orchestrated by Kubernetes, with different scaling characteristics based on the edition.

Core Application Services​

1. Coginiti Application Container​

  • Service Names: coginiti-team, coginiti-enterprise
  • Function: Main application server hosting the Coginiti web interface and API
  • Port: 8080 (internal)
  • Dependencies: PostgreSQL, Redis, Elasticsearch/OpenSearch, Artemis MQ, Embedded Service
  • Scaling: Single instance (Team) | 3+ replicas (Enterprise)

2. Traefik Reverse Proxy​

  • Service Name: traefik
  • Function: Load balancer and reverse proxy with SSL termination
  • Port: 443 (HTTPS external access)
  • Features:
    • Sticky sessions for user affinity
    • Automatic service discovery
    • SSL/TLS termination and certificate management
    • Request routing to application instances

Data Storage Services​

3. PostgreSQL Database​

  • Service Name: postgres
  • Container Image: public.ecr.aws/aginity/premium/aginity-db:15pgv
  • Function: Primary relational database for application data, metadata, and configuration
  • Port: 5432
  • Features: Health checks, persistent storage, pg_vector extension for AI capabilities
  • Scaling: Single instance with backup strategies

4. Redis Cache​

  • Service Name: redis
  • Container Image: public.ecr.aws/aginity/premium/aginity-cache:7.4.2
  • Function: In-memory caching for query results and session storage
  • Port: 6379
  • Scaling:
    • Team: Single instance
    • Enterprise: 6-node cluster (3 masters, 3 replicas)

5. Elasticsearch/OpenSearch​

  • Service Name: elasticsearch/opensearch
  • Container Image: public.ecr.aws/aginity/premium/aginity-search:2.19.2-slim
  • Function: Full-text search and analytics engine for catalog search and data discovery
  • Ports: 9200 (HTTP), 9300 (transport)
  • Scaling:
    • Team: Single-node deployment
    • Enterprise: 3-node cluster for high availability

Messaging & Communication​

6. Apache Artemis MQ​

  • Service Name: artemismq
  • Container Image: public.ecr.aws/aginity/premium/aginity-mq:2.41.0
  • Function: Message broker for asynchronous processing, job queuing, and inter-service communication
  • Port: 61616
  • Scaling:
    • Team: Single instance
    • Enterprise: 3-node cluster with high availability

AI & Analytics Services​

7. Embedded Service (AI/ML)​

  • Service Name: embedded-service
  • Container Image: public.ecr.aws/aginity/embedded:1.7.3
  • Function: AI/ML service for query optimization, code assistance, and intelligent data analysis
  • Port: 8000
  • Integration: Leverages pg_vector extension for vector operations

Utility Services​

8. Utility Service​

  • Container Image: public.ecr.aws/aginity/utility-image:1.6.7
  • Function: Database migrations, configuration tasks, maintenance operations, and system initialization

Security Architecture​

Authentication & Authorization​

Coginiti implements a multi-layered security model:

Identity Provider Integration​

  • SAML 2.0 and OAuth 2.0 support
  • Active Directory integration
  • Multi-factor authentication (MFA)
  • Single Sign-On (SSO)

Role-Based Access Control (RBAC)​

Fine-grained permissions at multiple levels:

  • Organization level
  • Project level
  • Connection level
  • Query level

Data Security​

Encryption​

  • In Transit: TLS 1.3 for all communications
  • At Rest: AES-256 encryption for stored data
  • Database Connections: Native database encryption support

Network Security​

  • VPC isolation for cloud deployments
  • Private networking options
  • IP allowlisting and geo-blocking
  • WAF (Web Application Firewall) protection

Performance & Scalability​

Query Execution​

Coginiti optimizes query performance through:

Connection Pooling​

Efficient database connection management reduces overhead and improves response times.

Query Caching​

Multi-tier caching strategy:

  • L1: Browser-level caching for UI elements
  • L2: Application-level caching for query results
  • L3: Database-level query plan caching

Async Processing​

Long-running queries are processed asynchronously, allowing users to continue working while queries execute in the background.

Auto-Scaling​

The platform automatically scales based on:

  • User activity levels
  • Query complexity and volume
  • Resource utilization metrics

Query Execution Architecture​

Multi-Stage Execution Pipeline​

Coginiti Team implements a sophisticated execution architecture designed for performance, scalability, and real-time result streaming.

1. Query Preparation Phase​

  • Script Analysis: SQL scripts are parsed and validated for syntax
  • Block Division: Complex scripts are divided into executable blocks
  • Parameter Collection: User parameters are collected for parameterized queries
  • Session Management: Database sessions are created and validated

2. Execution Dispatch​

  • Multi-User Architecture: Execution requests are distributed across dedicated execution servers
  • Load Balancing: Multiple execution servers handle concurrent user requests
  • Session Isolation: User sessions are managed independently for security and performance

3. Database Execution​

  • Platform Integration: Connections established to target database platforms
  • Parallel Processing: Multiple query blocks can execute concurrently
  • Connection Pooling: Database connections are efficiently reused

4. Result Streaming​

  • Real-Time Streaming: Results are streamed back to users as they become available
  • Multi-Part Protocol: Metadata, data, and errors are streamed as separate parts
  • Efficient Formats: Large result sets use Apache Arrow columnar format for performance
  • Progressive Updates: UI updates in real-time as results stream back

Communication Architecture​

Request/Response Flow​

  • HTTP REST API: Primary communication via /api/execute endpoint
  • Multipart Streaming: Results delivered via multipart HTTP responses
  • gRPC Communication: Internal service communication for scalability
  • WebSocket Support: Real-time updates and collaboration features

Data Formats​

  • JSON: Standard format for metadata and small result sets
  • Apache Arrow IPC: Columnar format for large datasets and analytics
  • Multipart MIME: Streaming protocol for progressive result delivery

Performance Optimizations​

Streaming & Caching​

  • Immediate Response: Results stream as soon as they're available
  • Memory Management: Client-side memory monitoring prevents browser crashes
  • Result Caching: Query results cached for reuse and performance
  • Connection Persistence: Database connections maintained between queries

Scalability Features​

  • Horizontal Scaling: Multiple execution servers handle increased load
  • Session Distribution: User sessions distributed across available servers
  • Resource Isolation: Users isolated from each other's resource usage
  • Automatic Cleanup: Sessions and resources automatically cleaned up

Error Handling & Recovery​

Robust Error Management​

  • Position Tracking: Errors include precise location information
  • Contextual Messages: Error messages provide execution context
  • Partial Results: System delivers partial results when possible
  • Graceful Degradation: System continues operating despite individual failures

Cancellation & Cleanup​

  • Query Cancellation: Users can cancel long-running queries
  • Resource Cleanup: Database connections and sessions properly cleaned up
  • Session Recovery: System recovers gracefully from connection failures

Data Flow Summary​

Web UI → REST API → Execution Servers → Database Platforms
  ↑         ↑           ↑                    ↓
  └─── Real-time ← Multipart ← Query Results ┘
     Result Updates    Streaming

This architecture ensures efficient handling of everything from simple queries to complex analytical workloads, providing real-time feedback and robust error handling throughout the execution process.

Deployment Configurations​

Team Edition Services​

Optimized for smaller teams with single-instance deployments:

  • Coginiti Team Application (single instance)
  • PostgreSQL (single instance)
  • Redis (single instance)
  • Elasticsearch/OpenSearch (single node)
  • Apache Artemis MQ (single instance)
  • Embedded Service (AI/ML capabilities)
  • Traefik (reverse proxy and SSL termination)

Enterprise Edition Services​

Designed for high availability and scalability:

  • Coginiti Enterprise Application (3+ replicas with load balancing)
  • PostgreSQL (single instance with enhanced backup strategies)
  • Redis Cluster (6 nodes: 3 masters + 3 replicas)
  • OpenSearch Cluster (3 nodes for high availability)
  • Apache Artemis MQ Cluster (3 nodes with HA)
  • Embedded Service (AI/ML capabilities)
  • Traefik (reverse proxy and SSL termination)

Additional Enterprise Components​

Monitoring & Logging​

  • AWS CloudWatch Agent: Metrics collection and monitoring
  • Fluent Bit: Log aggregation and forwarding to external systems
  • Grafana: Optional metrics visualization and dashboards

Service Dependencies & Communication Flow​

External Users (HTTPS:443)
        ↓
    Traefik (Reverse Proxy)
        ↓
Coginiti Application (8080) ←→ Embedded Service (8000)
        ↓
    ┌─── PostgreSQL (5432) - Primary data storage
    ├─── Redis (6379) - Caching and sessions  
    ├─── OpenSearch (9200/9300) - Search and analytics
    └─── Artemis MQ (61616) - Async messaging

Inter-Service Communication​

  • Traefik → Coginiti Application: HTTP/HTTPS request routing
  • Coginiti Application → PostgreSQL: Primary data operations and metadata storage
  • Coginiti Application → Redis: Session management and query result caching
  • Coginiti Application → OpenSearch: Full-text search and data catalog operations
  • Coginiti Application → Artemis MQ: Asynchronous job processing and messaging
  • Coginiti Application → Embedded Service: AI/ML query optimization and assistance

Resource Requirements​

Minimum Team Edition Deployment​

  • Server: 8 cores, 32GB RAM, 256GB storage
  • Application Container: 4GB memory allocation
  • PostgreSQL: 4GB memory allocation
  • Redis: 1GB memory allocation
  • OpenSearch: 2GB memory allocation
  • Artemis MQ: 1GB memory allocation

Enterprise Edition Resource Requirements​

  • Application Pods: 10GB memory each (3+ replicas)
  • PostgreSQL: 4GB memory minimum
  • Redis Cluster: 2-3GB memory per node (6 nodes total)
  • OpenSearch Cluster: 4GB memory per node (3 nodes)
  • Artemis MQ Cluster: 1.5-3GB memory per node (3 nodes)
  • Total Enterprise Minimum: 24+ cores, 96GB+ RAM

Cloud Instance Recommendations​

  • AWS: m5.2xlarge or m7i.2xlarge (8 vCPUs, 32 GiB RAM)
  • Azure: Standard_D8s_v3 (8 vCPUs, 32 GiB RAM)
  • GCP: n2-standard-8 (8 vCPUs, 32 GB RAM)

Deployment Models​

Cloud Marketplace Deployments​

AWS Marketplace: Pre-configured CloudFormation deployment with automatic infrastructure provisioning

Google Cloud Marketplace: BigQuery-certified deployment optimized for Google Cloud infrastructure

Self-Hosted Installation​

Direct installation on organization-managed infrastructure:

  • Complete control over server specifications
  • Custom networking and security configurations
  • Integration with existing enterprise systems
  • Compliance with specific regulatory requirements

Architectural Design Principles​

Coginiti's architecture is guided by key design principles that ensure the platform delivers reliable, scalable, and user-focused data analytics capabilities.

Adaptability and Modularity​

The platform is built with modular components that can evolve independently. This architectural flexibility allows Coginiti to adapt to new database technologies, analytical requirements, and organizational needs without requiring major system overhauls.

Streaming-First Performance​

Real-time data streaming is prioritized throughout the architecture. Results flow to users immediately as they become available, rather than waiting for complete query execution, enabling faster analysis and decision-making.

Transparent and Predictable Behavior​

The system avoids hidden "magic" behaviors. Users can understand and predict how features will work, leading to more confident and efficient workflows. Error messages are clear and actionable.

Scalable Resource Management​

Whether supporting individual analysts or large enterprise teams, the architecture scales efficiently:

  • Connection pooling ensures database resources are shared optimally
  • Session isolation provides security without sacrificing performance
  • Horizontal scaling accommodates growing user bases and data volumes

Security by Design​

Security considerations are integrated into every architectural layer:

  • Encryption protects data in transit and at rest
  • Role-based access control manages permissions at multiple levels
  • Session management isolates user activities securely

Separation of Concerns​

Clear architectural boundaries separate different system responsibilities:

  • Presentation layer handles user interface and interaction
  • Application layer manages business logic and query orchestration
  • Data layer handles storage, caching, and external system integration
  • Infrastructure layer provides scalable, reliable service hosting

For a deeper understanding of how these principles guide user experience and platform capabilities, see the Design Principles explanation.

This architectural approach ensures Coginiti can adapt to changing requirements while maintaining performance, security, and reliability at scale.

Implementation Guides​

Understanding Coginiti​