Skip to main content

Coginiti Actions Reference

Complete reference for Coginiti Actions configuration syntax and options.

File Structure​

Coginiti Actions are defined in TOML (Tom's Obvious Minimal Language) format with this structure:

[general]
# Workflow metadata

[job.job_name]  
# Job configuration

[job.another_job]
# Another job configuration

General Section​

The [general] section contains workflow-level metadata.

Required Fields​

name​

  • Type: String
  • Required: Yes
  • Description: Unique identifier for the Coginiti action
  • Example: name = "daily_sales_pipeline"

Optional Fields​

description​

  • Type: String
  • Required: No
  • Description: Human-readable description of the workflow's purpose
  • Example: description = "Process daily sales data for reporting"

General Section Example​

[general]
name = "customer_etl_pipeline"
description = "Extract, transform, and load customer data from multiple sources"

Job Sections​

Each job is defined as [job.job_name] where job_name is a unique identifier.

Required Fields​

steps​

  • Type: Array of strings
  • Required: Yes
  • Description: List of step names to execute in sequential order
  • Example: steps = ["extract_data", "validate_data", "transform_data"]

Optional Fields​

depends_on​

  • Type: Array of strings
  • Required: No
  • Description: List of job names that must complete successfully before this job runs
  • Example: depends_on = ["extract_job", "validation_job"]

Job Section Example​

[job.data_processing]
depends_on = ["data_extraction"]
steps = ["clean_data", "validate_data", "enrich_data"]

Step Definitions​

Each step referenced in the steps array must have a corresponding definition.

Step Structure​

step_name = { command = "command_type", asset = "asset_path", parameters = { } }

Required Step Fields​

command​

  • Type: String
  • Required: Yes
  • Valid Values: "run_asset"
  • Description: The command type to execute

asset​

  • Type: String
  • Required: Yes
  • Description: Path to the SQL asset in the catalog
  • Format: Absolute path starting with /
  • Example: asset = "/reports/daily_summary"

Optional Step Fields​

parameters​

  • Type: Object (key-value pairs)
  • Required: No
  • Description: Parameters to pass to the SQL asset
  • Format: { "$param_name" = "value" }

Step Definition Examples​

Simple Step​

extract_data = { command = "run_asset", asset = "/etl/extract" }

Step with Parameters​

daily_report = { command = "run_asset", asset = "/reports/daily", parameters = { "$report_date" = "2024-01-15", "$department" = "sales", "$limit" = "1000" } }

Parameter Syntax​

Parameters enable dynamic value substitution in SQL assets.

Parameter Format​

  • Naming: Must start with $ followed by parameter name
  • SQL Usage: Reference in SQL as '$parameter_name'
  • TOML Definition: "$parameter_name" = "value"

Parameter Types​

String Parameters​

parameters = { "$customer_name" = "Acme Corp" }

Date Parameters​

parameters = { "$start_date" = "2024-01-01", "$end_date" = "2024-01-31" }

Numeric Parameters​

parameters = { "$threshold" = "100", "$limit" = "5000" }

Dynamic Parameters​

Omit parameters from TOML to make them dynamic:

# This will prompt for parameters during execution
process_data = { command = "run_asset", asset = "/process/data" }

Dependency Rules​

Valid Dependencies​

  • Job names must exactly match other job names in the same file
  • Dependencies cannot be circular (A depends on B, B depends on A)
  • A job can depend on multiple other jobs
  • Multiple jobs can depend on the same job

Dependency Examples​

Single Dependency​

[job.transform]
depends_on = ["extract"]

Multiple Dependencies​

[job.report]
depends_on = ["sales_data", "customer_data", "inventory_data"]

Chain Dependencies​

[job.extract]
steps = ["extract"]

[job.transform] 
depends_on = ["extract"]
steps = ["transform"]

[job.load]
depends_on = ["transform"]  
steps = ["load"]

Complete Configuration Examples​

Simple Linear Pipeline​

[general]
name = "simple_etl"
description = "Basic ETL pipeline for customer data"

[job.extract]
steps = ["extract_customers"]
extract_customers = { command = "run_asset", asset = "/etl/extract_customers" }

[job.transform]
depends_on = ["extract"]
steps = ["transform_customers"]
transform_customers = { command = "run_asset", asset = "/etl/transform_customers" }

[job.load]
depends_on = ["transform"]
steps = ["load_customers"]
load_customers = { command = "run_asset", asset = "/etl/load_customers" }

Parallel Processing Pipeline​

[general]
name = "parallel_processing"
description = "Process multiple data sources in parallel"

[job.process_sales]
steps = ["extract_sales", "transform_sales"]
extract_sales = { command = "run_asset", asset = "/data/extract_sales" }
transform_sales = { command = "run_asset", asset = "/data/transform_sales" }

[job.process_customers]
steps = ["extract_customers", "transform_customers"]
extract_customers = { command = "run_asset", asset = "/data/extract_customers" }
transform_customers = { command = "run_asset", asset = "/data/transform_customers" }

[job.generate_report]
depends_on = ["process_sales", "process_customers"]
steps = ["create_report"]
create_report = { command = "run_asset", asset = "/reports/combined_report" }

Complex Multi-Branch Pipeline​

[general]
name = "complex_analytics"
description = "Multi-stage analytics pipeline with quality checks"

[job.data_ingestion]
steps = ["ingest_raw_data", "initial_validation"]
ingest_raw_data = { command = "run_asset", asset = "/ingestion/ingest" }
initial_validation = { command = "run_asset", asset = "/quality/initial_check" }

[job.sales_processing]
depends_on = ["data_ingestion"]
steps = ["process_sales", "validate_sales"]
process_sales = { command = "run_asset", asset = "/processing/sales" }
validate_sales = { command = "run_asset", asset = "/quality/sales_validation" }

[job.customer_processing]
depends_on = ["data_ingestion"]
steps = ["process_customers", "validate_customers"]
process_customers = { command = "run_asset", asset = "/processing/customers" }
validate_customers = { command = "run_asset", asset = "/quality/customer_validation" }

[job.data_enrichment]
depends_on = ["sales_processing", "customer_processing"]
steps = ["enrich_data", "calculate_metrics"]
enrich_data = { command = "run_asset", asset = "/enrichment/enrich" }
calculate_metrics = { command = "run_asset", asset = "/metrics/calculate" }

[job.quality_assurance]
depends_on = ["data_enrichment"]
steps = ["final_validation", "data_profiling"]
final_validation = { command = "run_asset", asset = "/quality/final_check" }
data_profiling = { command = "run_asset", asset = "/quality/profile" }

[job.reporting]
depends_on = ["quality_assurance"]
steps = ["executive_summary", "detailed_report", "data_export"]
executive_summary = { command = "run_asset", asset = "/reports/executive" }
detailed_report = { command = "run_asset", asset = "/reports/detailed" }
data_export = { command = "run_asset", asset = "/export/final_export" }

Syntax Rules​

TOML Formatting​

  • Section headers use square brackets: [section]
  • Job sections use dot notation: [job.job_name]
  • Strings must be quoted: name = "value"
  • Arrays use square brackets: depends_on = ["job1", "job2"]

Naming Conventions​

  • Job names: Use lowercase with underscores (data_processing)
  • Step names: Use lowercase with underscores (extract_data)
  • Parameter names: Start with $ ($report_date)

Required Quotation​

All string values must be quoted:

# Correct
name = "my_workflow"
asset = "/path/to/script"

# Incorrect  
name = my_workflow
asset = /path/to/script

Validation Rules​

Configuration Validation​

  • General section must contain name field
  • Each job must have a steps array
  • All steps in steps array must have definitions
  • All depends_on references must point to existing jobs
  • No circular dependencies allowed

Runtime Validation​

  • All referenced assets must exist in the catalog
  • User must have access to all referenced assets
  • Parameter values must match SQL parameter requirements
  • Database connections must be available

Error Reference​

Common Configuration Errors​

Missing Required Field​

Error: Job 'data_processing' is missing required field 'steps'

Invalid Dependency​

Error: Job 'transform' depends on non-existent job 'extact' (did you mean 'extract'?)

Circular Dependency​

Error: Circular dependency detected: job_a -> job_b -> job_a

Step Definition Missing​

Error: Step 'process_data' referenced in job 'processing' but not defined

Common Runtime Errors​

Asset Not Found​

Error: Asset '/etl/missing_script' not found in catalog

Parameter Mismatch​

Error: SQL asset expects parameter '$report_date' but none provided

Access Denied​

Error: User does not have permission to execute asset '/restricted/script'

Best Practices​

Organization​

  • Use descriptive job and step names
  • Group related steps into logical jobs
  • Keep job dependencies simple and clear

Parameters​

  • Define parameters in TOML for static values
  • Leave parameters undefined for dynamic scheduling
  • Use consistent parameter naming across workflows

Error Handling​

  • Validate configuration before scheduling
  • Test workflows with sample data
  • Monitor execution logs for errors

Performance​

  • Avoid unnecessary dependencies that prevent parallelization
  • Keep individual steps focused and efficient
  • Use appropriate indexing in SQL assets