Connecting to Object Stores

This guide explains how to configure connections to cloud object storage services in Coginiti. Object store connections allow you to browse, upload, download, and manage files stored in Amazon S3, Azure Blob Storage, and Google Cloud Storage directly from within Coginiti.

Overview

Coginiti supports connections to major cloud object storage services, enabling you to:

• Browse storage buckets and containers directly in the application
• Upload and download files to/from cloud storage
• Manage file organization with folder structures
• Access data files for analysis and processing
• Store query results and exports in the cloud

Supported Object Store Types

• Amazon S3 - AWS Simple Storage Service
• MinIO - S3-compatible object storage for on-premises and private cloud
• Azure Blob Storage - Microsoft Azure blob containers
• Google Cloud Storage - Google Cloud Platform storage buckets

Prerequisites

General Requirements

• Coginiti Team or Enterprise with object store feature enabled
• Cloud storage account with appropriate service configured
• Access credentials with required permissions for each service
• Network connectivity from Coginiti to cloud storage endpoints

Permission Requirements

Each cloud provider requires specific permissions for Coginiti to access and manage your storage resources. See platform-specific sections below for detailed permission lists.

Creating Object Store Connections

Accessing the Connection Dialog

To create a new object store connection:

1. Use the keyboard shortcut ⌘+U (Mac) or Ctrl+U (Windows/Linux)
2. Or navigate to Connections → Add Connection → Object Store
3. The Object Store Connection dialog will appear

Quick Access

The ⌘+U keyboard shortcut provides the fastest way to access object store connection configuration from anywhere in the application.

Amazon S3 Configuration

Amazon S3 is AWS's highly scalable object storage service, providing industry-standard security, performance, and durability for data storage and retrieval.

Connection Settings

Configure the following fields for Amazon S3:

Basic Configuration

• Connection Name: Choose a unique and meaningful name (e.g., "Production S3", "Analytics Data Lake")
• Object Store Type: Select "Amazon S3" from the dropdown
• Access Key ID: Your AWS access key identifier
• Secret Access Key: Your AWS secret access key
• Region: AWS region where your buckets are located

Optional Configuration

• Buckets: Comma-separated list of specific buckets to display (leave empty to show all accessible buckets)

Example S3 Configuration

Connection Name: Production Data Lake
Object Store Type: Amazon S3
Access Key ID: AKIAIOSFODNN7EXAMPLE
Secret Access Key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Region: us-east-1
Buckets: analytics-data, reports-archive, user-uploads

AWS Permissions Required

Configure the following IAM permissions for the user or role associated with your access keys:

Minimum Required Permissions

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::your-bucket-name",
                "arn:aws:s3:::your-bucket-name/*"
            ]
        }
    ]
}

Full Access Permissions

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation",
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::your-bucket-name",
                "arn:aws:s3:::your-bucket-name/*"
            ]
        }
    ]
}

Permission Descriptions

• s3:ListBucket: Browse bucket contents and folder structures
• s3:GetBucketLocation: Determine bucket region and properties
• s3:GetObject: Download and read files from buckets
• s3:PutObject: Upload files to buckets
• s3:DeleteObject: Delete files and folders from buckets

Multi-Region S3 Setup

If you need to work with buckets across multiple AWS regions:

Create Separate Connections

Connection 1:
Name: "S3 US East"
Region: us-east-1
Buckets: east-analytics, east-logs

Connection 2:
Name: "S3 EU West"  
Region: eu-west-1
Buckets: eu-analytics, eu-reports

Benefits of Multi-Region Connections

• Optimal performance: Connect to buckets in their native regions
• Reduced latency: Faster access to geographically distributed data
• Cost optimization: Avoid cross-region data transfer charges
• Compliance: Meet data residency requirements

S3 Security Best Practices

Access Key Management

• Use IAM users with minimal required permissions
• Rotate access keys regularly (every 90 days recommended)
• Never share access keys between users or applications
• Consider IAM roles for enhanced security

Bucket Security

• Enable bucket versioning for data protection
• Configure bucket encryption for sensitive data
• Use bucket policies to restrict access patterns
• Enable CloudTrail logging for audit trails

MinIO Configuration

MinIO is an S3-compatible object storage server that can be deployed on-premises or in private cloud environments, providing high-performance storage with the same API as Amazon S3.

Connection Settings

Configure the following fields for MinIO:

Basic Configuration

• Connection Name: Choose a unique and meaningful name (e.g., "On-Premises MinIO", "Private Cloud Storage")
• Object Store Type: Select "MinIO" or "S3-Compatible" from the dropdown
• Endpoint URL: Your MinIO server endpoint (e.g., https://minio.company.com:9000)
• Access Key: Your MinIO access key
• Secret Key: Your MinIO secret key
• Region: MinIO region (can be custom region name)

Optional Configuration

• Buckets: Comma-separated list of specific buckets to display (leave empty to show all accessible buckets)

Example MinIO Configuration

Connection Name: Corporate MinIO
Object Store Type: MinIO
Endpoint URL: https://minio.company.com:9000
Access Key: minioadmin
Secret Key: minioadmin123
Region: us-east-1
Buckets: analytics-data, backups, shared-files

MinIO Permissions Required

MinIO uses the same permission model as Amazon S3:

Policy Configuration

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation",
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::your-bucket-name",
                "arn:aws:s3:::your-bucket-name/*"
            ]
        }
    ]
}

User Management

Create dedicated users for Coginiti integration:

# Create MinIO user
mc admin user add myminio coginiti-user SecurePassword123

# Create policy file (save as coginiti-policy.json)
# Apply policy to user
mc admin policy add myminio coginiti-policy coginiti-policy.json
mc admin policy set myminio coginiti-policy user=coginiti-user

MinIO Security Best Practices

Access Control

• Create dedicated service users for application integration
• Use strong passwords for MinIO user accounts
• Implement bucket policies for fine-grained access control
• Enable TLS/SSL for encrypted communication

Network Security

• Configure reverse proxy (Nginx, Apache) for additional security
• Implement firewall rules to restrict access to MinIO ports
• Use VPN or private networks for enhanced security
• Monitor access logs for suspicious activity

High Availability

• Deploy MinIO in cluster mode for production environments
• Configure distributed storage across multiple nodes
• Implement backup strategies for critical data
• Monitor cluster health and performance metrics

MinIO vs AWS S3 Comparison

Feature	MinIO	AWS S3
Deployment	On-premises/Private cloud	AWS cloud service
API Compatibility	Full S3 API compatibility	Native S3 API
Cost	Hardware + operational costs	Pay-per-use pricing
Control	Full infrastructure control	AWS-managed service
Scalability	Manual scaling required	Automatic scaling
Security	Self-managed security	AWS security infrastructure

Azure Blob Storage Configuration

Connection Settings

Configure the following fields for Azure Blob Storage:

Basic Configuration

• Connection Name: Choose a unique and meaningful name (e.g., "Production Blob Storage", "Analytics Archive")
• Object Store Type: Select "Azure Blob Storage" from the dropdown
• Storage Account Name: Your Azure storage account name
• Key: Your Azure storage account access key

Optional Configuration

• Blob Containers: Comma-separated list of specific containers to display (leave empty to show all accessible containers)

Example Azure Blob Configuration

Connection Name: Corporate Data Storage
Object Store Type: Azure Blob Storage
Storage Account Name: companydata
Key: Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==
Blob Containers: analytics, reports, backups

Azure Permissions Required

Configure the following permissions for your storage account:

Access Key Permissions

When using storage account access keys, you automatically have:

• Full access to all storage account resources
• All container operations: create, list, delete containers
• All blob operations: upload, download, delete blobs
• Account management: view account properties and metrics

Role-Based Access Control (RBAC)

For more granular control, consider these Azure roles:

Storage Blob Data Reader:

• Read and list containers and blobs
• Download blob content

Storage Blob Data Contributor:

• Read, write, and delete containers and blobs
• Upload and modify blob content

Storage Blob Data Owner:

• Full access including access control management
• All blob and container operations

Azure Security Considerations

Access Key Security

• Regenerate keys regularly (quarterly recommended)
• Use key rotation to maintain service availability
• Monitor key usage through Azure Monitor
• Consider SAS tokens for limited-time access

Container Security

• Configure container access levels appropriately
• Use Azure Active Directory for authentication when possible
• Enable soft delete for blob recovery
• Implement lifecycle management for cost optimization

Google Cloud Storage Configuration

Connection Settings

Configure the following fields for Google Cloud Storage:

Basic Configuration

• Connection Name: Choose a unique and meaningful name (e.g., "GCS Data Lake", "Analytics Storage")
• Object Store Type: Select "Google Cloud Storage" from the dropdown
• Authentication Type: Choose between "Service Account" or "User Account"
• Private Key: Upload your JSON service account key file (click folder icon)
• Project ID: Your Google Cloud project identifier

Optional Configuration

• Buckets: Comma-separated list of specific buckets to display (leave empty to show all accessible buckets)

Example GCS Configuration

Connection Name: Analytics Data Platform
Object Store Type: Google Cloud Storage
Authentication: Service Account
Private Key: [service-account-key.json file]
Project ID: company-analytics-prod
Buckets: raw-data, processed-data, ml-models

Authentication Methods

Service Account Authentication (Recommended)

Benefits:

• Programmatic access with JSON key files
• Fine-grained permission control
• Independent of user accounts
• Suitable for production environments

Setup Process:

1. Create service account in Google Cloud Console
2. Generate JSON key file for the service account
3. Assign appropriate roles to the service account
4. Upload JSON file in Coginiti connection dialog

User Account Authentication

Benefits:

• Uses personal Google account credentials
• OAuth-based authentication flow
• Suitable for development and testing
• Inherits user's existing permissions

Setup Process:

1. Select "User Account" authentication type
2. Complete OAuth flow when prompted
3. Grant necessary permissions to Coginiti
4. Verify access to required buckets

Google Cloud Permissions Required

Configure the following IAM permissions for your service account or user:

Minimum Required Permissions

# Basic read-only access
storage.buckets.get
storage.buckets.list
storage.objects.list
storage.objects.get

Full Access Permissions

# Complete file management capabilities
storage.buckets.get
storage.buckets.list
storage.objects.list
storage.objects.get
storage.objects.create
storage.objects.delete

IAM Role Assignments

Storage Object Viewer:

• storage.objects.get
• storage.objects.list

Storage Object Admin:

• All object operations including create and delete
• Full bucket content management

Storage Admin:

• Complete storage management including bucket operations
• Project-level storage administration

GCS Security Best Practices

Service Account Security

• Use dedicated service accounts for Coginiti integration
• Apply principle of least privilege to service account roles
• Rotate service account keys regularly
• Monitor service account usage through Cloud Logging

Bucket Security

• Configure bucket permissions with IAM policies
• Enable uniform bucket-level access for consistent security
• Use Cloud KMS for encryption key management
• Implement retention policies for compliance

Managing Multiple Object Store Connections

Connection Organization

Naming Conventions

Use consistent naming patterns for easy identification:

Environment-Provider-Purpose:
- "PROD-S3-DataLake"
- "DEV-Azure-TestData"  
- "STAGING-GCS-Analytics"

Department-Provider-Region:
- "Finance-S3-USEast"
- "Marketing-Azure-Europe"
- "Engineering-GCS-Asia"

Connection Grouping

Organize connections by:

• Environment (Production, Staging, Development)
• Department (Finance, Marketing, Engineering)
• Data Type (Raw Data, Processed Data, Archives)
• Geographic Region (US, Europe, Asia-Pacific)

Multi-Credential Management

Separate Connections for Different Access Levels

Read-Only Connection:
- Name: "S3-Analytics-ReadOnly"
- Permissions: s3:GetObject, s3:ListBucket
- Users: All analysts and viewers

Read-Write Connection:
- Name: "S3-Analytics-FullAccess"
- Permissions: Full S3 permissions
- Users: Data engineers and administrators

Project-Based Connections

Project Alpha:
- S3: alpha-raw-data, alpha-processed
- GCS: alpha-ml-models, alpha-results
- Azure: alpha-backups, alpha-archives

Project Beta:
- S3: beta-datasets, beta-exports
- GCS: beta-training-data
- Azure: beta-compliance-data

Using Object Store Connections

Browsing Storage

Navigation Interface

Once connected, you can:

• Browse bucket/container hierarchy in the connection panel
• Navigate folder structures by expanding directories
• View file properties including size, modification date, and type
• Search for specific files using built-in search functionality

File Operations

Download Files:

• Right-click on files to download to local system
• Select multiple files for batch downloads
• Choose download location and maintain folder structure

Upload Files:

• Drag and drop files from local system
• Use upload dialog for file selection
• Create folders and organize uploaded content

File Management:

• Rename files and folders (if permissions allow)
• Delete files and folders (with appropriate permissions)
• Move files between folders within the same connection

Integration with Coginiti Features

Query Results Export

• Export query results directly to object storage
• Choose export format (CSV, JSON, Parquet, etc.)
• Specify destination path within connected buckets
• Schedule automated exports for regular data updates

Data Import and Analysis

• Import data files from object storage for analysis
• Preview file contents before importing
• Configure import parameters (delimiters, headers, data types)
• Create data connections to files for ongoing analysis

Troubleshooting Object Store Connections

Common Connection Issues

Authentication Failures

Symptoms: Cannot connect to object store service

Solutions for AWS S3:

1. Verify access key and secret: Check for typos and correct values
2. Test AWS CLI access: Use same credentials with AWS CLI
3. Check IAM permissions: Ensure all required permissions are granted
4. Verify region settings: Confirm buckets exist in specified region

Solutions for Azure Blob:

1. Verify storage account name: Check spelling and case sensitivity
2. Regenerate access key: Try with newly generated access key
3. Test with Azure CLI: Verify credentials work with Azure tools
4. Check firewall rules: Ensure Coginiti IP is allowed

Solutions for Google Cloud:

1. Validate service account key: Ensure JSON file is valid and complete
2. Check project ID: Verify project exists and is accessible
3. Test with gcloud CLI: Use same service account with gcloud tools
4. Verify IAM roles: Confirm service account has required permissions

Permission Errors

Symptoms: Can browse but cannot upload/download files

Solutions:

1. Review permission requirements for your use case
2. Test specific operations with cloud provider tools
3. Check bucket policies and access restrictions
4. Verify credential scope includes required permissions

Network Connectivity Issues

Symptoms: Timeouts or connection refused errors

Solutions:

1. Check internet connectivity from Coginiti server
2. Verify firewall rules allow outbound HTTPS traffic
3. Test DNS resolution for cloud storage endpoints
4. Configure proxy settings if required for your network

Performance Optimization

Connection Performance

Optimize Settings:

• Choose nearest regions for best performance
• Limit bucket lists to reduce connection overhead
• Use connection pooling for frequent operations
• Configure appropriate timeouts for your network conditions

File Transfer Optimization

Best Practices:

• Use parallel uploads/downloads for large files
• Compress files before uploading when appropriate
• Implement resume capabilities for interrupted transfers
• Monitor transfer speeds and adjust configurations

Security Best Practices

Credential Security

Access Key Management

Best Practices:

• Use dedicated credentials for Coginiti integration
• Implement regular key rotation (90-day cycles recommended)
• Store credentials securely within Coginiti
• Monitor credential usage through cloud provider logs

Principle of Least Privilege

Implementation:

• Grant minimum required permissions for intended use cases
• Use read-only access when write operations aren't needed
• Restrict bucket access to only required containers/buckets
• Review permissions regularly and remove unnecessary access

Network Security

Connection Encryption

• Use HTTPS/TLS for all object store communications
• Verify SSL certificates to prevent man-in-the-middle attacks
• Configure secure cipher suites when possible
• Monitor for certificate expiration

Access Control

• Implement IP whitelisting where supported
• Use VPC endpoints for enhanced security (AWS)
• Configure private endpoints (Azure)
• Enable private Google access (GCP)

Compliance Considerations

Data Governance

Implement Controls:

• Data classification and labeling
• Access logging and audit trails
• Data retention policies and lifecycle management
• Compliance reporting and monitoring

Regulatory Requirements

Consider Regulations:

• GDPR compliance for European data
• HIPAA requirements for healthcare data
• SOX compliance for financial data
• Industry-specific regulations as applicable

Summary

You have successfully configured object store connections for Coginiti! Key achievements:

✅ Multi-Cloud Support: Connections to AWS S3, Azure Blob Storage, and Google Cloud Storage ✅ Secure Authentication: Proper credential configuration and permission management ✅ File Operations: Browse, upload, download, and manage cloud storage files ✅ Integration: Seamless integration with Coginiti query and analysis features ✅ Security: Implementation of best practices for credential and data security ✅ Multi-Connection Management: Organization of multiple storage connections

Your Coginiti instance now has comprehensive cloud storage integration, enabling efficient data management and analysis workflows across multiple cloud platforms.