Skip to content

Market Data Collection

This document describes the market data collection functionality implemented in DSTA, including BTC dominance, total market cap, Fear & Greed Index, and order book data.

Overview

The market data module provides comprehensive tools for collecting and analyzing market-wide metrics and order book data. This enables context-aware trading decisions based on broader market conditions and real-time order flow.

Features

1. BTC Dominance Tracking

What it is: Bitcoin's market capitalization as a percentage of the total cryptocurrency market cap.

Why it matters: BTC dominance is a key indicator of Bitcoin's relative strength. High dominance suggests investors are favoring Bitcoin over altcoins, while low dominance indicates altcoin strength.

Data Source: CoinGecko API

Update Frequency: Recommended: Every 15-60 minutes

2. Total Market Cap Tracking

What it is: The total market capitalization of all cryptocurrencies.

Why it matters: Total market cap reflects overall market size and health. Growth indicates capital flowing into crypto, while decline suggests capital outflow.

Data Source: CoinGecko API

Update Frequency: Recommended: Every 15-60 minutes

3. Fear & Greed Index

What it is: A sentiment indicator ranging from 0 (Extreme Fear) to 100 (Extreme Greed).

Why it matters: The index helps identify market extremes. Extreme Fear often presents buying opportunities, while Extreme Greed suggests caution.

Scale: - 0-24: Extreme Fear - 25-49: Fear - 50: Neutral - 51-75: Greed - 76-100: Extreme Greed

Data Source: Alternative.me API

Update Frequency: Updated daily

Enhanced Features: - Historical data storage (up to 365 days) - Trend analysis - Extreme value alerts - Classification tracking

4. Order Book Data

What it is: Real-time snapshots of bids and asks at various price levels (L2 market depth).

Why it matters: Order book analysis reveals: - Support and resistance levels (large volume walls) - Market sentiment (buy vs sell pressure) - Potential price movements - Liquidity at different price levels

Supported Exchanges: Binance (more exchanges coming soon)

Update Frequency: Real-time to every few seconds

Analysis Features: - Support/resistance level detection - Order book imbalance calculation - Spread metrics - Aggregated depth visualization

Installation & Setup

Prerequisites

Ensure you have the required dependencies:

pip install requests numpy django djangorestframework

For exchange order book data, you'll also need:

pip install python-binance

Configuration

Set up API credentials in your .env file:

# For order book collection from Binance
BINANCE_API_KEY=your_api_key_here
BINANCE_API_SECRET=your_api_secret_here

Usage

Command Line Interface

Collect Market Metrics

Collect all market metrics (BTC dominance, market cap, Fear & Greed Index):

python src/manage.py collect_market_metrics

Collect specific metrics:

# Only BTC dominance
python src/manage.py collect_market_metrics --metrics btc_dominance

# BTC dominance and market cap
python src/manage.py collect_market_metrics --metrics btc_dominance market_cap

# Fear & Greed with 90 days of history
python src/manage.py collect_market_metrics --metrics fear_greed --fear-greed-days 90

Analyze Fear & Greed Index trend:

python src/manage.py collect_market_metrics --metrics fear_greed --analyze-trend

Check for extreme values:

python src/manage.py collect_market_metrics --metrics fear_greed --check-extremes \
  --extreme-fear-threshold 20 --extreme-greed-threshold 80

Collect Order Book Data

Collect order book snapshots:

# Single collection for BTCUSDT
python src/manage.py collect_orderbooks --symbols BTCUSDT --depth 100

# Multiple symbols
python src/manage.py collect_orderbooks --symbols BTCUSDT ETHUSDT BNBUSDT --depth 100

# With analysis display
python src/manage.py collect_orderbooks --symbols BTCUSDT --depth 100 --analyze

# Continuous collection (every 5 seconds, 10 times)
python src/manage.py collect_orderbooks --symbols BTCUSDT \
  --depth 100 --interval 5 --count 10

Python API

Market Metrics Collection

from api.tasks.market_metrics import (
    MarketMetricsCollector,
    collect_and_store_btc_dominance,
    collect_and_store_total_market_cap,
    collect_and_store_fear_greed_index,
)

# Initialize collector
collector = MarketMetricsCollector()

# Collect BTC dominance
btc_dom = collector.get_btc_dominance()
print(f"BTC Dominance: {btc_dom['value']}%")

# Collect total market cap
market_cap = collector.get_total_market_cap()
print(f"Total Market Cap: ${market_cap['value']:,.0f}")

# Collect Fear & Greed Index (last 7 days)
fear_greed = collector.get_fear_greed_index(limit=7)
for entry in fear_greed:
    print(f"{entry['timestamp']}: {entry['value']} - {entry['metadata']['value_classification']}")

# Analyze trend
trend = collector.analyze_fear_greed_trend(days=30)
print(f"Current: {trend['current_value']}, Trend: {trend['trend']}")

# Check for extremes
alert = collector.check_extreme_values()
if alert['is_extreme']:
    print(alert['message'])

# Store directly in database
collect_and_store_btc_dominance()
collect_and_store_total_market_cap()
collect_and_store_fear_greed_index(limit=30)

Order Book Collection

from api.orderbook import OrderBookCollector, OrderBookAnalyzer
from api.exchanges.binance_client import BinanceClient

# Initialize exchange client
client = BinanceClient(api_key='your_key', api_secret='your_secret')

# Initialize collector
collector = OrderBookCollector(client)

# Fetch and analyze order book
data = collector.fetch_and_analyze('BTCUSDT', limit=100)

# Access analysis results
print(f"Best Bid: {data['spread']['best_bid']}")
print(f"Best Ask: {data['spread']['best_ask']}")
print(f"Spread: {data['spread']['spread_percentage']:.4f}%")
print(f"Order Book Pressure: {data['imbalance']['pressure']}")
print(f"Support Levels: {data['support_resistance']['support'][:3]}")
print(f"Resistance Levels: {data['support_resistance']['resistance'][:3]}")

# Standalone analyzer
analyzer = OrderBookAnalyzer()
bids = [[50000, 1.5], [49999, 2.0], [49998, 1.8]]
asks = [[50001, 1.3], [50002, 2.2], [50003, 1.5]]

# Calculate support/resistance
sr = analyzer.calculate_support_resistance(bids, asks)
print(f"Support: {sr['support']}")
print(f"Resistance: {sr['resistance']}")

# Calculate imbalance
imbalance = analyzer.calculate_order_book_imbalance(bids, asks)
print(f"Imbalance: {imbalance['imbalance_percentage']:.2f}%")
print(f"Pressure: {imbalance['pressure']}")

# Calculate spread
spread = analyzer.calculate_spread_metrics(bids, asks)
print(f"Mid Price: {spread['mid_price']}")
print(f"Spread: {spread['spread_percentage']:.4f}%")

REST API Endpoints

Get Market Metrics

GET /api/market/metrics/

Query parameters: - metric_type: Filter by type (btc_dominance, total_market_cap, fear_greed_index) - start_date: Start date (ISO 8601 format) - end_date: End date (ISO 8601 format) - limit: Number of results (default: 100, max: 1000)

Example:

# Get last 50 BTC dominance values
curl "http://localhost:8000/api/market/metrics/?metric_type=btc_dominance&limit=50"

# Get Fear & Greed Index for last 7 days
curl "http://localhost:8000/api/market/metrics/?metric_type=fear_greed_index&limit=7"

Response:

{
  "count": 2,
  "results": [
    {
      "id": 1,
      "metric_type": "btc_dominance",
      "timestamp": "2025-11-21T16:00:00Z",
      "value": "45.50",
      "metadata": {
        "total_market_cap_usd": 2000000000000.0,
        "active_cryptocurrencies": 13000
      },
      "created_at": "2025-11-21T16:05:00Z"
    }
  ]
}

Get Order Book Snapshots

GET /api/market/orderbooks/

Query parameters: - exchange: Filter by exchange - symbol: Filter by symbol - start_date: Start date (ISO 8601 format) - end_date: End date (ISO 8601 format) - limit: Number of results (default: 10, max: 100)

Example:

# Get last 10 BTCUSDT order book snapshots from Binance
curl "http://localhost:8000/api/market/orderbooks/?exchange=binance&symbol=BTCUSDT&limit=10"

Response:

{
  "count": 1,
  "results": [
    {
      "id": 1,
      "exchange": "binance",
      "symbol": "BTCUSDT",
      "timestamp": "2025-11-21T16:00:00Z",
      "bids": [[50000.0, 1.5], [49999.0, 2.0]],
      "asks": [[50001.0, 1.3], [50002.0, 2.2]],
      "last_update_id": 12345,
      "created_at": "2025-11-21T16:00:05Z"
    }
  ]
}

Scheduled Collection with Celery

To automate market data collection, set up Celery tasks:

# In src/api/tasks/celery_tasks.py
from celery import shared_task
from api.tasks.market_metrics import (
    collect_and_store_btc_dominance,
    collect_and_store_total_market_cap,
    collect_and_store_fear_greed_index,
)

@shared_task
def collect_market_metrics_task():
    """Collect all market metrics."""
    collect_and_store_btc_dominance()
    collect_and_store_total_market_cap()
    collect_and_store_fear_greed_index(limit=1)

@shared_task
def collect_order_book_task(symbol='BTCUSDT'):
    """Collect order book snapshot."""
    from api.orderbook import collect_and_store_order_book
    from api.exchanges.binance_client import BinanceClient
    from dsta.config import Config

    config = Config()
    client = BinanceClient(
        api_key=config.get('BINANCE_API_KEY'),
        api_secret=config.get('BINANCE_API_SECRET')
    )

    collect_and_store_order_book(client, symbol)

Configure periodic tasks in settings.py:

from celery.schedules import crontab

CELERY_BEAT_SCHEDULE = {
    'collect-market-metrics-every-30-minutes': {
        'task': 'api.tasks.celery_tasks.collect_market_metrics_task',
        'schedule': crontab(minute='*/30'),
    },
    'collect-orderbook-every-5-seconds': {
        'task': 'api.tasks.celery_tasks.collect_order_book_task',
        'schedule': 5.0,  # seconds
        'args': ('BTCUSDT',)
    },
}

Data Analysis Examples

Identifying Market Conditions

from api.models import MarketMetric
from datetime import datetime, timedelta

# Get recent metrics
now = datetime.now()
week_ago = now - timedelta(days=7)

btc_dom = MarketMetric.objects.filter(
    metric_type='btc_dominance',
    timestamp__gte=week_ago
).order_by('-timestamp').first()

fear_greed = MarketMetric.objects.filter(
    metric_type='fear_greed_index',
    timestamp__gte=week_ago
).order_by('-timestamp').first()

# Analyze conditions
if btc_dom and fear_greed:
    btc_value = float(btc_dom.value)
    fg_value = float(fear_greed.value)

    if fg_value < 25 and btc_value > 50:
        print("📊 Market Condition: EXTREME FEAR with BTC dominance rising")
        print("💡 Strategy: Consider accumulating quality altcoins")

    elif fg_value > 75 and btc_value < 40:
        print("📊 Market Condition: EXTREME GREED with low BTC dominance")
        print("💡 Strategy: Consider taking profits on altcoins")

Order Book Analysis for Entry/Exit

from api.orderbook import OrderBookCollector
from api.exchanges.binance_client import BinanceClient

client = BinanceClient()
collector = OrderBookCollector(client)

# Analyze current order book
data = collector.fetch_and_analyze('BTCUSDT', limit=100)

imbalance = data['imbalance']
support = data['support_resistance']['support']
resistance = data['support_resistance']['resistance']

# Decision logic
if imbalance['pressure'] == 'buying' and imbalance['imbalance_percentage'] > 20:
    print("🟢 Strong buying pressure detected")
    print(f"   First support level: ${support[0]:,.2f}")
    print(f"   Suggested entry: Around ${support[0]:,.2f}")

elif imbalance['pressure'] == 'selling' and imbalance['imbalance_percentage'] < -20:
    print("🔴 Strong selling pressure detected")
    print(f"   First resistance level: ${resistance[0]:,.2f}")
    print(f"   Suggested exit: Around ${resistance[0]:,.2f}")

Best Practices

Collection Frequency

  • BTC Dominance & Market Cap: Every 15-60 minutes (API rate limits apply)
  • Fear & Greed Index: Once per day (updates daily)
  • Order Book: Real-time to every 5 seconds (for active trading)

Storage Considerations

  • Order book data can grow large quickly. Consider:
  • Storing only top 20-50 levels instead of full depth
  • Aggregating older data
  • Implementing data retention policies

Error Handling

All collection functions include error handling and logging. Monitor logs for:

import logging
logger = logging.getLogger('api.tasks.market_metrics')
logger.setLevel(logging.INFO)

Rate Limiting

  • CoinGecko: 10-50 requests per minute (free tier)
  • Alternative.me: No strict limits but be respectful
  • Binance: 1200 requests per minute (weight-based)

Troubleshooting

Common Issues

Issue: API requests failing

# Check connectivity
curl https://api.coingecko.com/api/v3/ping
curl https://api.alternative.me/fng/

Issue: Database connection errors

# Check Django settings and database status
python src/manage.py dbshell

Issue: Order book data not updating

  • Verify API credentials are correct
  • Check exchange client connection: client.ping()
  • Ensure exchange client is properly initialized

Debugging

Enable debug logging:

import logging
logging.basicConfig(level=logging.DEBUG)

Future Enhancements

  • Add more exchanges for order book data (Coinbase, Kraken, etc.)
  • Implement WebSocket streams for real-time order book updates
  • Add more market metrics (funding rates, open interest, liquidations)
  • Create visualization dashboard
  • Implement anomaly detection for market metrics
  • Add historical data backfilling

References

Support

For issues or questions: - Create an issue on GitHub - Check existing documentation in /docs - Review test files for usage examples

License

This module is part of the DSTA project and follows the same license.