Setting up Logging and Monitoring

This guide explains how to access, manage, and integrate Coginiti Team logs with external monitoring and log aggregation systems. Coginiti Team logs all application system events to the local filesystem for comprehensive monitoring and troubleshooting.

Overview

Coginiti Team provides comprehensive logging of all application system events, making it easy to monitor system health, troubleshoot issues, and integrate with enterprise logging infrastructure.

Coginiti maintains two distinct types of logs:

Application Logs - System operations, performance metrics, and technical events
Security Audit Logs - User activities, authentication events, and compliance tracking

Security Audit Logs

Security audit logs are separate from application logs and have their own format and structure. For detailed information about security events and audit logging, see the Security Log Reference.

Log Storage Location

All Coginiti Team logs are stored in the standard Linux log location:

/var/log/containers/

This location follows Linux logging conventions and makes it easy to integrate with existing log management tools and processes. Both application logs and security audit logs are stored in this location but in separate files with different naming patterns.

Log File Structure

Log File Organization

Coginiti Team organizes logs by component and type:

Application Logs

/var/log/containers/
├── coginiti-app-*.log          # Main application logs
├── coginiti-web-*.log          # Web server logs  
├── coginiti-auth-*.log         # Authentication service logs
├── coginiti-db-*.log           # Database connection logs
├── coginiti-worker-*.log       # Background worker logs
└── coginiti-system-*.log       # System-level logs

Security Audit Logs

/var/log/containers/
├── coginiti-security-*.log     # Security audit events (CEF format)
├── coginiti-audit-*.log        # User activity audit trails
└── coginiti-compliance-*.log   # Compliance and regulatory events

Log Type Identification

Application logs focus on system operations and performance, while security audit logs track user activities and security events in CEF (Common Event Format). Both types are essential for comprehensive monitoring and compliance.

Log File Naming Convention

Log files follow a consistent naming pattern:

Format: coginiti-[component]-[timestamp].log
Rotation: New files created daily or when size limits are reached
Retention: Configurable retention period (default: 30 days)

Log Entry Format

Coginiti Team uses structured logging with the following format:

{
  "timestamp": "2024-07-30T16:45:23.123Z",
  "level": "INFO",
  "component": "coginiti-app",
  "message": "User authentication successful",
  "user_id": "user@company.com",
  "session_id": "sess_abc123",
  "request_id": "req_xyz789",
  "metadata": {
    "duration_ms": 245,
    "ip_address": "192.168.1.100"
  }
}

Log Categories and Content

This section covers application logs used for system monitoring and troubleshooting. For security audit logs, see the Security Log Reference.

Application Logs (`coginiti-app-*.log`)

System startup and shutdown events
Query execution and performance metrics
Data source connections and operations
Feature usage and analytics
Error conditions and exceptions
Resource utilization metrics

Web Server Logs (`coginiti-web-*.log`)

HTTP request/response logging
API endpoint access
Static resource serving
Load balancer health checks
SSL certificate status
Performance metrics

Authentication Service Logs (`coginiti-auth-*.log`)

OAuth/OIDC authentication flows (technical aspects)
Session management events
Token validation and refresh
Service-to-service authentication
Authentication system performance

Security vs Application Authentication Logs

These authentication service logs focus on technical system operations. User login attempts, security events, and audit trails are recorded in separate security audit logs. See the Security Log Reference for user activity tracking.

Database Logs (`coginiti-db-*.log`)

Database connection establishment/termination
Query execution times and results
Connection pool status
Database schema changes
Performance metrics

Worker Logs (`coginiti-worker-*.log`)

Background job processing
Scheduled task execution
Data refresh operations
System maintenance tasks
Resource cleanup activities

System Logs (`coginiti-system-*.log`)

Container startup/shutdown events
Resource utilization metrics
Health check results
Configuration changes
System alerts and warnings

Accessing Logs

Direct File System Access

Access logs directly on the Coginiti Team server:

# View recent application logs
sudo tail -f /var/log/containers/coginiti-app-*.log

# Search for specific events
sudo grep "authentication" /var/log/containers/coginiti-auth-*.log

# View logs from the last hour
sudo find /var/log/containers/ -name "coginiti-*.log" -newermt "1 hour ago" -exec tail {} \;

Using Standard Linux Log Tools

journalctl Integration

If Coginiti Team is running as a systemd service:

# View Coginiti service logs
sudo journalctl -u coginiti-team -f

# View logs from specific time period
sudo journalctl -u coginiti-team --since "2024-07-30 10:00:00"

logrotate Configuration

Ensure proper log rotation to prevent disk space issues:

# Check logrotate configuration
sudo cat /etc/logrotate.d/coginiti-team

# Manually trigger log rotation
sudo logrotate -f /etc/logrotate.d/coginiti-team

Log Management Best Practices

Log Retention Configuration

Configure appropriate log retention based on your requirements:

Storage Requirements

Development: 7-14 days retention
Production: 30-90 days retention
Compliance: May require 1+ years retention

Disk Space Planning

# Monitor log directory size
du -sh /var/log/containers/

# Check available disk space
df -h /var/log/

Log Rotation Setup

Configure automatic log rotation to prevent disk space issues:

# Create logrotate configuration
sudo tee /etc/logrotate.d/coginiti-team << EOF
/var/log/containers/coginiti-*.log {
    daily
    rotate 30
    compress
    delaycompress
    missingok
    notifempty
    create 644 coginiti coginiti
    postrotate
        # Send HUP signal to application to reopen log files
        pkill -HUP -f coginiti-team
    endscript
}
EOF

Monitoring Log Health

Set up monitoring to ensure logging is working correctly:

# Check if logs are being written
sudo ls -la /var/log/containers/ | head -10

# Monitor log growth
sudo watch -n 60 'du -sh /var/log/containers/'

# Check for log errors
sudo grep -i "error\|exception\|failed" /var/log/containers/coginiti-*.log | tail -20

Integration with External Systems

Graylog Integration

Configure Graylog to collect Coginiti Team logs:

1. Install Graylog Sidecar

# Download and install Graylog Sidecar
wget https://github.com/Graylog2/collector-sidecar/releases/download/1.4.0/graylog-sidecar_1.4.0-1_amd64.deb
sudo dpkg -i graylog-sidecar_1.4.0-1_amd64.deb

2. Configure Sidecar

# Edit sidecar configuration
sudo tee /etc/graylog/sidecar/sidecar.yml << EOF
server_url: "http://your-graylog-server:9000/api/"
server_api_token: "your-api-token"
node_id: "coginiti-team-server"
node_name: "Coginiti Team Logs"
tags: ["coginiti", "application"]
log_path: "/var/log/graylog-sidecar"
EOF

3. Configure Filebeat Input

Create a Graylog input configuration for Coginiti logs:

# Filebeat configuration for Coginiti logs
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/containers/coginiti-*.log
  fields:
    application: coginiti-team
    environment: production
  fields_under_root: true
  json.keys_under_root: true
  json.add_error_key: true

output.gelf:
  hosts: ["your-graylog-server:12201"]

Logstash Integration

Configure Logstash to process Coginiti Team logs:

1. Logstash Input Configuration

# /etc/logstash/conf.d/coginiti-input.conf
input {
  file {
    path => "/var/log/containers/coginiti-*.log"
    start_position => "beginning"
    tags => ["coginiti", "application"]
    codec => "json"
  }
}

2. Logstash Filter Configuration

# /etc/logstash/conf.d/coginiti-filter.conf
filter {
  if "coginiti" in [tags] {
    # Parse timestamp
    date {
      match => [ "timestamp", "ISO8601" ]
    }
  
    # Add environment fields
    mutate {
      add_field => { "application" => "coginiti-team" }
      add_field => { "environment" => "production" }
    }
  
    # Parse log levels
    if [level] == "ERROR" or [level] == "FATAL" {
      mutate {
        add_tag => ["alert"]
      }
    }
  }
}

3. Logstash Output Configuration

# /etc/logstash/conf.d/coginiti-output.conf
output {
  if "coginiti" in [tags] {
    elasticsearch {
      hosts => ["your-elasticsearch-cluster:9200"]
      index => "coginiti-logs-%{+YYYY.MM.dd}"
    }
  }
}

ELK Stack Integration

Complete ELK (Elasticsearch, Logstash, Kibana) integration:

1. Elasticsearch Index Template

{
  "index_patterns": ["coginiti-logs-*"],
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 1,
    "index.refresh_interval": "30s"
  },
  "mappings": {
    "properties": {
      "timestamp": {
        "type": "date",
        "format": "strict_date_optional_time||epoch_millis"
      },
      "level": {
        "type": "keyword"
      },
      "component": {
        "type": "keyword"
      },
      "message": {
        "type": "text",
        "analyzer": "standard"
      },
      "user_id": {
        "type": "keyword"
      },
      "request_id": {
        "type": "keyword"
      }
    }
  }
}

2. Kibana Dashboard Configuration

Create Kibana visualizations for:

Log level distribution over time
Component-specific error rates
User activity patterns
System performance metrics
Authentication events

Splunk Integration

Configure Splunk Universal Forwarder:

1. Install Splunk Universal Forwarder

# Download and install Splunk Universal Forwarder
wget -O splunkforwarder.tgz "https://download.splunk.com/products/universalforwarder/releases/9.1.0/linux/splunkforwarder-9.1.0-linux-x86_64.tgz"
sudo tar xzf splunkforwarder.tgz -C /opt/

2. Configure Inputs

# /opt/splunkforwarder/etc/system/local/inputs.conf
[monitor:///var/log/containers/coginiti-*.log]
disabled = false
host = coginiti-team-server
index = coginiti
sourcetype = coginiti:application

3. Configure Outputs

# /opt/splunkforwarder/etc/system/local/outputs.conf
[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = your-splunk-indexer:9997

Monitoring and Alerting

Key Metrics to Monitor

Application Health

Error rate by component
Response time percentiles
System resource utilization
Query execution performance
Service availability

System Resources

Log file growth rate
Disk space utilization
Memory usage patterns
CPU utilization
Network connectivity

Security Events (from Security Audit Logs)

For security-related monitoring, focus on the security audit logs:

Failed authentication attempts
Unusual access patterns
Permission escalation events
Data export activities

See the Security Log Reference for detailed security event monitoring and the specific CEF format fields available for alerting.

Sample Monitoring Queries

Application Log Queries

Graylog Queries:

# High error rate detection
component:coginiti-app AND level:ERROR

# System performance issues
component:coginiti-system AND level:WARN

# Slow query detection
component:coginiti-db AND duration_ms:>5000

# Service availability monitoring
component:coginiti-web AND level:ERROR

Security Audit Log Queries

For security monitoring queries using CEF format fields, see the Security Log Reference which includes specific examples for:

Failed authentication detection
User activity monitoring
Data access auditing
Compliance reporting

Logstash/Elasticsearch Queries

{
  "query": {
    "bool": {
      "must": [
        {"term": {"component": "coginiti-app"}},
        {"term": {"level": "ERROR"}},
        {"range": {"timestamp": {"gte": "now-1h"}}}
      ]
    }
  }
}

Alerting Configuration

Set Up Alerts For:

Critical Errors: Application exceptions and system failures
Security Events: Multiple failed logins, unauthorized access attempts
Performance Issues: Slow queries, high response times
System Health: Disk space, memory usage, service availability

Sample Alert Rules

Graylog Alert Example:

# Alert when error rate exceeds threshold
component:coginiti-app AND level:ERROR
# Condition: More than 10 messages in the last 5 minutes

Elasticsearch Watcher Example:

{
  "trigger": {
    "schedule": {"interval": "5m"}
  },
  "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": ["coginiti-logs-*"],
        "body": {
          "query": {
            "bool": {
              "must": [
                {"term": {"level": "ERROR"}},
                {"range": {"timestamp": {"gte": "now-5m"}}}
              ]
            }
          }
        }
      }
    }
  },
  "condition": {
    "compare": {"ctx.payload.hits.total": {"gt": 10}}
  }
}

Troubleshooting

Common Log Issues

Log Files Not Being Created

Symptoms: No log files in /var/log/containers/

Solutions:

Check Coginiti Team service status:
```
sudo systemctl status coginiti-team
```

Verify directory permissions:

sudo ls -la /var/log/containers/
sudo chown -R coginiti:coginiti /var/log/containers/

Check disk space:
```
df -h /var/log/
```

Log Files Growing Too Quickly

Symptoms: Rapid disk space consumption

Solutions:

Implement log rotation:

sudo logrotate -f /etc/logrotate.d/coginiti-team

Adjust log level configuration
Increase log rotation frequency

Missing Log Entries

Symptoms: Expected events not appearing in logs

Solutions:

Check log level configuration
Verify component-specific logging settings
Check for log buffering issues

Integration Troubleshooting

Graylog Not Receiving Logs

Check Graylog Sidecar status:
```
sudo systemctl status graylog-sidecar
```
Verify network connectivity to Graylog server
Check Graylog input configuration

Logstash Processing Issues

Check Logstash service status:
```
sudo systemctl status logstash
```

Verify configuration syntax:

sudo /usr/share/logstash/bin/logstash --config.test_and_exit

Monitor Logstash logs for errors

Security Considerations

Log Security Best Practices

Access Control

Restrict log file access to authorized users only
Use appropriate file permissions (640 or 644)
Implement log aggregation system authentication

Sensitive Data Protection

Ensure logs don't contain passwords or tokens
Implement log data encryption in transit
Consider log data encryption at rest

Compliance Requirements

Configure retention policies per regulatory requirements
Implement audit trails for log access
Ensure log integrity and tamper-proofing

Sample Security Configuration

# Set secure permissions on log directory
sudo chown -R coginiti:coginiti /var/log/containers/
sudo chmod -R 640 /var/log/containers/

# Configure logrotate with secure permissions
sudo tee -a /etc/logrotate.d/coginiti-team << EOF
    create 640 coginiti coginiti
EOF

Performance Optimization

Log Performance Tuning

Reduce Log Volume

Configure appropriate log levels for each environment
Use sampling for high-volume debug logs
Implement conditional logging for performance-sensitive operations

Optimize I/O Performance

Use dedicated disk/partition for logs
Configure async logging where possible
Implement log buffering to reduce write frequency

Network Optimization

Use compression for log shipping
Implement batching for remote log transmission
Configure appropriate network timeouts

Summary

You have successfully configured logging and monitoring for Coginiti Team! Key achievements:

✅ Log Access: Understanding of log locations and file structure ✅ Log Management: Proper rotation, retention, and monitoring setup ✅ External Integration: Configuration with popular log aggregation systems ✅ Monitoring: Key metrics and alerting for system health ✅ Security: Proper access controls and sensitive data protection ✅ Troubleshooting: Common issues and resolution procedures

Your Coginiti Team instance now has comprehensive logging capabilities that support operational monitoring, troubleshooting, and compliance requirements.

Overview​

Log Storage Location​

Log File Structure​

Log File Organization​

Application Logs​

Security Audit Logs​

Log File Naming Convention​

Log Entry Format​

Log Categories and Content​

Application Logs (coginiti-app-*.log)​

Web Server Logs (coginiti-web-*.log)​

Authentication Service Logs (coginiti-auth-*.log)​

Database Logs (coginiti-db-*.log)​

Worker Logs (coginiti-worker-*.log)​

System Logs (coginiti-system-*.log)​

Accessing Logs​

Direct File System Access​

Using Standard Linux Log Tools​

journalctl Integration​

logrotate Configuration​

Log Management Best Practices​

Log Retention Configuration​

Storage Requirements​

Disk Space Planning​

Log Rotation Setup​

Monitoring Log Health​

Integration with External Systems​

Graylog Integration​

1. Install Graylog Sidecar​

2. Configure Sidecar​

3. Configure Filebeat Input​

Logstash Integration​

1. Logstash Input Configuration​

2. Logstash Filter Configuration​

3. Logstash Output Configuration​

ELK Stack Integration​

1. Elasticsearch Index Template​

2. Kibana Dashboard Configuration​

Splunk Integration​

1. Install Splunk Universal Forwarder​

2. Configure Inputs​

3. Configure Outputs​

Monitoring and Alerting​

Key Metrics to Monitor​

Application Health​

System Resources​

Security Events (from Security Audit Logs)​

Sample Monitoring Queries​

Application Log Queries​

Security Audit Log Queries​

Logstash/Elasticsearch Queries​

Alerting Configuration​

Set Up Alerts For:​

Sample Alert Rules​

Troubleshooting​

Common Log Issues​

Log Files Not Being Created​

Log Files Growing Too Quickly​

Missing Log Entries​

Integration Troubleshooting​

Graylog Not Receiving Logs​

Logstash Processing Issues​

Security Considerations​

Log Security Best Practices​

Access Control​

Sensitive Data Protection​

Compliance Requirements​

Sample Security Configuration​

Performance Optimization​

Log Performance Tuning​

Reduce Log Volume​

Optimize I/O Performance​

Network Optimization​

Summary​

Overview

Log Storage Location

Log File Structure

Log File Organization

Application Logs

Security Audit Logs

Log File Naming Convention

Log Entry Format

Log Categories and Content

Application Logs (`coginiti-app-*.log`)

Web Server Logs (`coginiti-web-*.log`)

Authentication Service Logs (`coginiti-auth-*.log`)

Database Logs (`coginiti-db-*.log`)

Worker Logs (`coginiti-worker-*.log`)

System Logs (`coginiti-system-*.log`)

Accessing Logs

Direct File System Access

Using Standard Linux Log Tools

journalctl Integration

logrotate Configuration

Log Management Best Practices

Log Retention Configuration

Storage Requirements

Disk Space Planning

Log Rotation Setup

Monitoring Log Health

Integration with External Systems

Graylog Integration

1. Install Graylog Sidecar

2. Configure Sidecar

3. Configure Filebeat Input

Logstash Integration

1. Logstash Input Configuration

2. Logstash Filter Configuration

3. Logstash Output Configuration

ELK Stack Integration

1. Elasticsearch Index Template

2. Kibana Dashboard Configuration

Splunk Integration

1. Install Splunk Universal Forwarder

2. Configure Inputs

3. Configure Outputs

Monitoring and Alerting

Key Metrics to Monitor

Application Health

System Resources

Security Events (from Security Audit Logs)

Sample Monitoring Queries

Application Log Queries

Security Audit Log Queries

Logstash/Elasticsearch Queries

Alerting Configuration

Set Up Alerts For:

Sample Alert Rules

Troubleshooting

Common Log Issues

Log Files Not Being Created

Log Files Growing Too Quickly

Missing Log Entries

Integration Troubleshooting

Graylog Not Receiving Logs

Logstash Processing Issues

Security Considerations

Log Security Best Practices

Access Control

Sensitive Data Protection

Compliance Requirements

Sample Security Configuration

Performance Optimization

Log Performance Tuning

Reduce Log Volume

Optimize I/O Performance

Network Optimization

Summary