Skip to content

CI/CD Pipeline Documentation

Overview

The DSTA project uses GitHub Actions for continuous integration and continuous deployment (CI/CD). The pipeline automatically runs on every push to main, develop, and copilot branches, as well as on pull requests.

Pipeline Components

1. Code Linting

Ensures code quality and consistency through automated checks:

  • Black: Code formatting
  • Line length: 120 characters
  • Python 3.12 target

  • Configuration: pyproject.toml under [tool.black]

  • isort: Import statement sorting

  • Profile: black (compatible with black formatter)

  • Configuration: pyproject.toml under [tool.isort]

  • Flake8: Code style and quality checks

  • Max line length: 120
  • Max complexity: 10

  • Configuration: .flake8

  • mypy: Static type checking (optional)

  • Python 3.12
  • Ignores missing imports for third-party libraries
  • Configuration: pyproject.toml under [tool.mypy]

2. Security Scanning

Identifies security vulnerabilities in code and dependencies:

  • Bandit: Python security linter
  • Scans for common security issues
  • Generates JSON reports for analysis

  • Reports are uploaded as artifacts

  • Safety: Dependency vulnerability checker

  • Checks installed packages against known CVE database
  • Alerts on outdated or vulnerable dependencies

3. Automated Testing

Runs comprehensive test suite:

  • Test Framework: pytest with pytest-django
  • Database: PostgreSQL 17 (service container)
  • Cache: Redis 8 (service container for future use)
  • Coverage: pytest-cov generates coverage reports

Test Markers: - unit: Unit tests (fast, isolated) - integration: Integration tests (slower, multiple components) - slow: Tests that take significant time - exchange: Tests requiring exchange API - database: Tests requiring database - redis: Tests requiring Redis - celery: Tests involving Celery tasks - websocket: Tests involving WebSocket connections

4. Docker Image Build (Optional)

Builds Docker images for deployment: - Only runs on pushes to main or develop branches - Uses Docker Buildx for efficient builds - Implements layer caching for faster builds - Tags images with branch name and commit SHA

Running Locally

Install Development Dependencies

# Install linting tools
pip install flake8 black isort mypy

# Install testing tools
pip install pytest pytest-django pytest-cov

# Install security tools
pip install bandit safety

Run Linters

# Format code with black
black src/ tests/

# Sort imports with isort
isort src/ tests/

# Check code quality with flake8
flake8 src/ tests/

# Run type checking with mypy
mypy src/

Run Tests

# Run all tests
pytest tests/

# Run with coverage
pytest tests/ --cov=src --cov-report=html

# Run specific test categories
pytest tests/ -m unit  # Only unit tests
pytest tests/ -m "not slow"  # Skip slow tests

Run Security Scans

# Scan code for security issues
bandit -r src/

# Check dependencies for vulnerabilities
safety check

Configuration Files

.flake8

Configures flake8 code linting: - Max line length: 120 - Max complexity: 10 - Excludes migrations, venv, etc. - Ignores errors that conflict with black

pyproject.toml

Central configuration for multiple tools: - [tool.black]: Black formatter settings - [tool.isort]: Import sorting settings - [tool.pytest.ini_options]: Pytest configuration - [tool.coverage.*]: Coverage reporting settings - [tool.mypy]: Type checking settings

pytest.ini

Pytest-specific configuration (legacy): - Test discovery patterns - Django settings module - Test markers - Output options

GitHub Actions Workflow

Location: .github/workflows/ci.yml

Trigger Events

  • Push to: main, develop, copilot/** branches
  • Pull requests to: main, develop
  • Manual workflow dispatch

Jobs

  1. lint: Runs all linting checks (black, isort, flake8)
  2. security: Runs security scans (bandit, safety)
  3. test: Runs test suite with coverage
  4. build-docker: Builds Docker image (main/develop only)
  5. summary: Aggregates results from all jobs

Job Dependencies

lint ──┐
       ├──> summary
test ──┤
       │
security ──┘

The build-docker job depends on all other jobs passing.

Best Practices

Before Committing

  1. Run linters locally: 

    black src/ tests/
isort src/ tests/
flake8 src/ tests/

  2. Run tests: 

    pytest tests/ -v

  3. Check security: 

    bandit -r src/

Writing Tests

  • Add appropriate markers (@pytest.mark.unit, etc.)
  • Mock external services (exchange APIs, etc.)
  • Use fixtures for common test data
  • Aim for high coverage (>80%) but prioritize meaningful tests

Handling CI Failures

  1. Linting Failures:
  2. Run the specific linter locally
  3. Fix formatting/style issues

  4. Commit and push

  5. Test Failures:

  6. Review test output in GitHub Actions
  7. Reproduce failure locally
  8. Fix the issue and add regression test

  9. Verify fix locally before pushing

  10. Security Failures:

  11. Review bandit/safety output
  12. Address legitimate security issues
  13. Add # nosec comment for false positives (with justification)
  14. Update vulnerable dependencies

Continuous Improvement

The CI/CD pipeline is continuously improved. Suggestions:

  1. Add code coverage thresholds (fail if < 80%)
  2. Add performance benchmarking
  3. Add automatic dependency updates (Dependabot)
  4. Add automatic changelog generation
  5. Add deployment to staging/production environments
  6. Add notification on failures (Slack, email)

Troubleshooting

Common Issues

Issue: Flake8 and Black disagree on formatting Solution: Black's formatting takes precedence. Update .flake8 to ignore conflicting rules.

Issue: Tests pass locally but fail in CI Solution: Usually environment-specific. Check: - Database connection settings - File paths (use absolute paths from project root) - Environment variables - Service availability (PostgreSQL, Redis)

Issue: Safety check fails on old dependencies Solution: Update dependencies in requirements.txt or add exceptions for false positives.

Resources