API Reference

This section provides detailed API documentation for all pyswark modules. The package is organized into layers, with each layer building on the ones below it.

Foundation Layer (Layer 0)

• lib
  • Key Features
  • Serialization (ser_des)
  • Base Models
  • Enum Utilities
  • AliasEnum
  • Filesystem (fsspec)
  • Usage Examples

Core Layer (Layer 1)

• core - I/O and Models
  • Key Features
  • I/O API
  • URI Schemes
  • Usage Examples
  • Core Models

Application Layer (Layer 2)

• gluedb - Data Catalogs
  • Key Concepts
  • GlueDb Class
  • GlueHub Class
  • Usage Examples
• workflow - Orchestration
  • Key Concepts
  • Workflow Class
  • Step Class
  • State Classes
  • Runner Class
  • Usage Examples
• tensor - Validated Arrays
  • Key Features
  • Classes
  • Usage Examples
• ts - Time Series
  • Key Features
  • Classes
  • Usage Examples
• sekrets - Credential Management
  • Key Concepts
  • API Functions
  • Usage Examples
  • Configuration
  • Security Best Practices
• query - Data Querying
  • Key Concepts
  • Classes
  • Usage Examples

Key principle: Layers only import from lower layers, never from higher layers.

Quick Import Guide

# Serialization (lib layer)
from pyswark.lib.pydantic import base, ser_des

# I/O operations (core layer)
from pyswark.core.io import api as io

# Data catalogs (application layer)
from pyswark.gluedb import api

# Workflows
from pyswark.workflow.workflow import Workflow
from pyswark.workflow.step import Step

# Tensors and time series
from pyswark.tensor.tensor import Vector, Matrix
from pyswark.ts.tsvector import TsVector