Logging Guide¶

This document describes the logging framework implemented in the DSTA project.

Overview¶

The DSTA project uses a comprehensive logging system with the following features:

Structured Logging: JSON format support for easy parsing and analysis
Log Rotation: Automatic rotation of log files to prevent disk space issues
Environment-Specific Levels: Different log levels for development, staging, and production
Request Tracking: Request ID tracking for debugging distributed systems
Separate Error Logs: Dedicated error log file for easy troubleshooting

Configuration¶

Environment Variables¶

Configure logging behavior using these environment variables:

ENVIRONMENT: Environment name (development, staging, production)
LOG_LEVEL: Explicit log level (DEBUG, INFO, WARNING, ERROR, CRITICAL)
LOG_DIR: Directory for log files (default: logs/)
USE_JSON_LOGGING: Enable JSON formatted logs (default: false)

Default Log Levels by Environment¶

Development: DEBUG
Staging: INFO
Production: WARNING

Log Files¶

The system creates the following log files in the log directory:

`dsta.log`¶

Main application log containing all log messages.

Max Size: 10 MB per file
Backup Count: 10 files (keeps up to 100 MB of logs)
Rotation: Automatic when file size exceeds limit

`error.log`¶

Dedicated error log containing only ERROR and CRITICAL level messages.

Max Size: 10 MB per file
Backup Count: 10 files
Rotation: Automatic when file size exceeds limit

Usage¶

Basic Logging¶

from dsta.logging_config import get_logger

logger = get_logger(__name__)

# Log messages
logger.debug("Debug information")
logger.info("Informational message")
logger.warning("Warning message")
logger.error("Error occurred")
logger.critical("Critical issue")

Logging with Context¶

from dsta.logging_config import get_context_logger

# Create logger with context
logger = get_context_logger(
    __name__,
    request_id='abc-123',
    user_id='user-456'
)

# Context will be automatically added to all log messages
logger.info("Processing request")

Logging Exceptions¶

logger = get_logger(__name__)

try:
    # Some code that might fail
    result = risky_operation()
except Exception as e:
    logger.error("Operation failed", exc_info=True)
    # exc_info=True includes full traceback

Logging in Django Views¶

from dsta.logging_config import get_logger

logger = get_logger(__name__)

def my_view(request):
    logger.info(
        "Processing view",
        extra={
            'request_id': request.request_id,  # Added by middleware
            'user': str(request.user)
        }
    )
    # View logic here

JSON Logging¶

Enable JSON logging for production environments where logs are ingested by log aggregation systems (ELK, Splunk, etc.).

Enable JSON Logging¶

export USE_JSON_LOGGING=true

JSON Log Format¶

{
  "timestamp": "2023-11-21T15:30:45.123456+00:00",
  "level": "INFO",
  "logger": "api.views",
  "message": "Request completed",
  "module": "views",
  "function": "my_view",
  "line": 42,
  "request_id": "abc-123-def-456",
  "user_id": "user-789"
}

JSON Log with Exception¶

{
  "timestamp": "2023-11-21T15:30:45.123456+00:00",
  "level": "ERROR",
  "logger": "api.views",
  "message": "Operation failed",
  "module": "views",
  "function": "my_view",
  "line": 42,
  "exception": "Traceback (most recent call last):\n  File \"views.py\", line 42...\nValueError: Invalid input"
}

Request Logging Middleware¶

The RequestLoggingMiddleware automatically logs all HTTP requests and responses.

Logged Information¶

Request Start: - Request ID (generated) - HTTP method - Path - Client IP address - User

Request Complete: - Request ID - HTTP method - Path - Status code - Duration (milliseconds) - User

Request Failed: - All of the above - Exception type - Full traceback

Request ID Header¶

The middleware adds an X-Request-ID header to all responses for tracking:

X-Request-ID: abc-123-def-456-ghi-789

Log Levels¶

DEBUG¶

Detailed diagnostic information. Only enabled in development.

Use for: - Variable values - Function entry/exit - Detailed flow information

INFO¶

General informational messages confirming normal operation.

Use for: - Request/response logging - Service startup/shutdown - Configuration values - Task completion

WARNING¶

Indication that something unexpected happened, but the application continues.

Use for: - Deprecated feature usage - Recoverable errors - Performance issues - Configuration problems

ERROR¶

Error that prevented a specific operation from completing.

Use for: - Failed API calls - Database errors - Failed validations - Exception handling

CRITICAL¶

Serious error that may prevent the application from continuing.

Use for: - Service unavailable - Database connection lost - Critical resource exhaustion

Best Practices¶

1. Use Appropriate Log Levels¶

# Good
logger.debug(f"Processing {len(items)} items")
logger.info("User logged in successfully")
logger.warning("Rate limit approaching")
logger.error("Failed to fetch data from API", exc_info=True)

# Bad
logger.info("x=5, y=10, z=15")  # Too detailed for INFO
logger.error("User not found")  # Not an error, should be WARNING or INFO

2. Include Context¶

# Good
logger.error(
    "Failed to process order",
    extra={
        'order_id': order.id,
        'user_id': order.user_id,
        'error': str(e)
    }
)

# Bad
logger.error("Failed to process order")

3. Don't Log Sensitive Data¶

# Bad - Never log passwords, API keys, tokens
logger.debug(f"Password: {password}")
logger.debug(f"API Key: {api_key}")

# Good - Log non-sensitive identifiers
logger.debug(f"Authenticating user: {username}")
logger.debug(f"Using API key ending in: {api_key[-4:]}")

4. Use Structured Logging¶

# Good - Structured
logger.info(
    "Order placed",
    extra={
        'order_id': 123,
        'user_id': 456,
        'amount': 99.99,
        'currency': 'USD'
    }
)

# Bad - Unstructured
logger.info(f"Order 123 placed by user 456 for $99.99 USD")

5. Log Exceptions Properly¶

# Good
try:
    result = risky_operation()
except Exception as e:
    logger.error("Operation failed", exc_info=True)
    # or
    logger.exception("Operation failed")  # Same as exc_info=True

# Bad
try:
    result = risky_operation()
except Exception as e:
    logger.error(f"Error: {str(e)}")  # Loses traceback

Third-Party Library Logging¶

The logging system automatically sets appropriate levels for common third-party libraries to reduce noise:

urllib3: WARNING
requests: WARNING
websockets: WARNING
aiohttp: WARNING

Testing¶

The logging framework includes comprehensive tests in tests/test_logging.py.

Run tests:

pytest tests/test_logging.py -v

Troubleshooting¶

Logs Not Appearing¶

Check log level: echo $LOG_LEVEL
Check log directory exists and is writable: ls -la logs/
Check Django settings include logging configuration

Log Files Too Large¶

Adjust rotation settings in logging_config.py:

file_handler = logging.handlers.RotatingFileHandler(
    main_log_file,
    maxBytes=5 * 1024 * 1024,  # Reduce to 5 MB
    backupCount=5,  # Keep fewer backup files
    encoding='utf-8'
)

Performance Issues¶

Reduce log level in production (WARNING or ERROR)
Disable console logging in production
Use asynchronous logging handler for high-volume applications

Example Configuration¶

Development¶

export ENVIRONMENT=development
export LOG_LEVEL=DEBUG
export LOG_DIR=logs
export USE_JSON_LOGGING=false

Production¶

export ENVIRONMENT=production
export LOG_LEVEL=WARNING
export LOG_DIR=/var/log/dsta
export USE_JSON_LOGGING=true

Integration with External Systems¶

ELK Stack (Elasticsearch, Logstash, Kibana)¶

Enable JSON logging and configure Filebeat to ship logs:

# filebeat.yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/dsta/*.log
  json.keys_under_root: true
  json.add_error_key: true

Splunk¶

Configure Splunk Universal Forwarder to monitor log directory:

[monitor:///var/log/dsta/]
disabled = false
sourcetype = json

CloudWatch Logs¶

Use AWS CloudWatch Logs agent to ship logs to AWS.