Market Data Collection¶
This document describes the market data collection functionality implemented in DSTA, including BTC dominance, total market cap, Fear & Greed Index, and order book data.
Overview¶
The market data module provides comprehensive tools for collecting and analyzing market-wide metrics and order book data. This enables context-aware trading decisions based on broader market conditions and real-time order flow.
Features¶
1. BTC Dominance Tracking¶
What it is: Bitcoin's market capitalization as a percentage of the total cryptocurrency market cap.
Why it matters: BTC dominance is a key indicator of Bitcoin's relative strength. High dominance suggests investors are favoring Bitcoin over altcoins, while low dominance indicates altcoin strength.
Data Source: CoinGecko API
Update Frequency: Recommended: Every 15-60 minutes
2. Total Market Cap Tracking¶
What it is: The total market capitalization of all cryptocurrencies.
Why it matters: Total market cap reflects overall market size and health. Growth indicates capital flowing into crypto, while decline suggests capital outflow.
Data Source: CoinGecko API
Update Frequency: Recommended: Every 15-60 minutes
3. Fear & Greed Index¶
What it is: A sentiment indicator ranging from 0 (Extreme Fear) to 100 (Extreme Greed).
Why it matters: The index helps identify market extremes. Extreme Fear often presents buying opportunities, while Extreme Greed suggests caution.
Scale: - 0-24: Extreme Fear - 25-49: Fear - 50: Neutral - 51-75: Greed - 76-100: Extreme Greed
Data Source: Alternative.me API
Update Frequency: Updated daily
Enhanced Features: - Historical data storage (up to 365 days) - Trend analysis - Extreme value alerts - Classification tracking
4. Order Book Data¶
What it is: Real-time snapshots of bids and asks at various price levels (L2 market depth).
Why it matters: Order book analysis reveals: - Support and resistance levels (large volume walls) - Market sentiment (buy vs sell pressure) - Potential price movements - Liquidity at different price levels
Supported Exchanges: Binance (more exchanges coming soon)
Update Frequency: Real-time to every few seconds
Analysis Features: - Support/resistance level detection - Order book imbalance calculation - Spread metrics - Aggregated depth visualization
Installation & Setup¶
Prerequisites¶
Ensure you have the required dependencies:
For exchange order book data, you'll also need:
Configuration¶
Set up API credentials in your .env file:
# For order book collection from Binance
BINANCE_API_KEY=your_api_key_here
BINANCE_API_SECRET=your_api_secret_here
Usage¶
Command Line Interface¶
Collect Market Metrics¶
Collect all market metrics (BTC dominance, market cap, Fear & Greed Index):
Collect specific metrics:
# Only BTC dominance
python src/manage.py collect_market_metrics --metrics btc_dominance
# BTC dominance and market cap
python src/manage.py collect_market_metrics --metrics btc_dominance market_cap
# Fear & Greed with 90 days of history
python src/manage.py collect_market_metrics --metrics fear_greed --fear-greed-days 90
Analyze Fear & Greed Index trend:
Check for extreme values:
python src/manage.py collect_market_metrics --metrics fear_greed --check-extremes \
--extreme-fear-threshold 20 --extreme-greed-threshold 80
Collect Order Book Data¶
Collect order book snapshots:
# Single collection for BTCUSDT
python src/manage.py collect_orderbooks --symbols BTCUSDT --depth 100
# Multiple symbols
python src/manage.py collect_orderbooks --symbols BTCUSDT ETHUSDT BNBUSDT --depth 100
# With analysis display
python src/manage.py collect_orderbooks --symbols BTCUSDT --depth 100 --analyze
# Continuous collection (every 5 seconds, 10 times)
python src/manage.py collect_orderbooks --symbols BTCUSDT \
--depth 100 --interval 5 --count 10
Python API¶
Market Metrics Collection¶
from api.tasks.market_metrics import (
MarketMetricsCollector,
collect_and_store_btc_dominance,
collect_and_store_total_market_cap,
collect_and_store_fear_greed_index,
)
# Initialize collector
collector = MarketMetricsCollector()
# Collect BTC dominance
btc_dom = collector.get_btc_dominance()
print(f"BTC Dominance: {btc_dom['value']}%")
# Collect total market cap
market_cap = collector.get_total_market_cap()
print(f"Total Market Cap: ${market_cap['value']:,.0f}")
# Collect Fear & Greed Index (last 7 days)
fear_greed = collector.get_fear_greed_index(limit=7)
for entry in fear_greed:
print(f"{entry['timestamp']}: {entry['value']} - {entry['metadata']['value_classification']}")
# Analyze trend
trend = collector.analyze_fear_greed_trend(days=30)
print(f"Current: {trend['current_value']}, Trend: {trend['trend']}")
# Check for extremes
alert = collector.check_extreme_values()
if alert['is_extreme']:
print(alert['message'])
# Store directly in database
collect_and_store_btc_dominance()
collect_and_store_total_market_cap()
collect_and_store_fear_greed_index(limit=30)
Order Book Collection¶
from api.orderbook import OrderBookCollector, OrderBookAnalyzer
from api.exchanges.binance_client import BinanceClient
# Initialize exchange client
client = BinanceClient(api_key='your_key', api_secret='your_secret')
# Initialize collector
collector = OrderBookCollector(client)
# Fetch and analyze order book
data = collector.fetch_and_analyze('BTCUSDT', limit=100)
# Access analysis results
print(f"Best Bid: {data['spread']['best_bid']}")
print(f"Best Ask: {data['spread']['best_ask']}")
print(f"Spread: {data['spread']['spread_percentage']:.4f}%")
print(f"Order Book Pressure: {data['imbalance']['pressure']}")
print(f"Support Levels: {data['support_resistance']['support'][:3]}")
print(f"Resistance Levels: {data['support_resistance']['resistance'][:3]}")
# Standalone analyzer
analyzer = OrderBookAnalyzer()
bids = [[50000, 1.5], [49999, 2.0], [49998, 1.8]]
asks = [[50001, 1.3], [50002, 2.2], [50003, 1.5]]
# Calculate support/resistance
sr = analyzer.calculate_support_resistance(bids, asks)
print(f"Support: {sr['support']}")
print(f"Resistance: {sr['resistance']}")
# Calculate imbalance
imbalance = analyzer.calculate_order_book_imbalance(bids, asks)
print(f"Imbalance: {imbalance['imbalance_percentage']:.2f}%")
print(f"Pressure: {imbalance['pressure']}")
# Calculate spread
spread = analyzer.calculate_spread_metrics(bids, asks)
print(f"Mid Price: {spread['mid_price']}")
print(f"Spread: {spread['spread_percentage']:.4f}%")
REST API Endpoints¶
Get Market Metrics¶
Query parameters: - metric_type: Filter by type (btc_dominance, total_market_cap, fear_greed_index) - start_date: Start date (ISO 8601 format) - end_date: End date (ISO 8601 format) - limit: Number of results (default: 100, max: 1000)
Example:
# Get last 50 BTC dominance values
curl "http://localhost:8000/api/market/metrics/?metric_type=btc_dominance&limit=50"
# Get Fear & Greed Index for last 7 days
curl "http://localhost:8000/api/market/metrics/?metric_type=fear_greed_index&limit=7"
Response:
{
"count": 2,
"results": [
{
"id": 1,
"metric_type": "btc_dominance",
"timestamp": "2025-11-21T16:00:00Z",
"value": "45.50",
"metadata": {
"total_market_cap_usd": 2000000000000.0,
"active_cryptocurrencies": 13000
},
"created_at": "2025-11-21T16:05:00Z"
}
]
}
Get Order Book Snapshots¶
Query parameters: - exchange: Filter by exchange - symbol: Filter by symbol - start_date: Start date (ISO 8601 format) - end_date: End date (ISO 8601 format) - limit: Number of results (default: 10, max: 100)
Example:
# Get last 10 BTCUSDT order book snapshots from Binance
curl "http://localhost:8000/api/market/orderbooks/?exchange=binance&symbol=BTCUSDT&limit=10"
Response:
{
"count": 1,
"results": [
{
"id": 1,
"exchange": "binance",
"symbol": "BTCUSDT",
"timestamp": "2025-11-21T16:00:00Z",
"bids": [[50000.0, 1.5], [49999.0, 2.0]],
"asks": [[50001.0, 1.3], [50002.0, 2.2]],
"last_update_id": 12345,
"created_at": "2025-11-21T16:00:05Z"
}
]
}
Scheduled Collection with Celery¶
To automate market data collection, set up Celery tasks:
# In src/api/tasks/celery_tasks.py
from celery import shared_task
from api.tasks.market_metrics import (
collect_and_store_btc_dominance,
collect_and_store_total_market_cap,
collect_and_store_fear_greed_index,
)
@shared_task
def collect_market_metrics_task():
"""Collect all market metrics."""
collect_and_store_btc_dominance()
collect_and_store_total_market_cap()
collect_and_store_fear_greed_index(limit=1)
@shared_task
def collect_order_book_task(symbol='BTCUSDT'):
"""Collect order book snapshot."""
from api.orderbook import collect_and_store_order_book
from api.exchanges.binance_client import BinanceClient
from dsta.config import Config
config = Config()
client = BinanceClient(
api_key=config.get('BINANCE_API_KEY'),
api_secret=config.get('BINANCE_API_SECRET')
)
collect_and_store_order_book(client, symbol)
Configure periodic tasks in settings.py:
from celery.schedules import crontab
CELERY_BEAT_SCHEDULE = {
'collect-market-metrics-every-30-minutes': {
'task': 'api.tasks.celery_tasks.collect_market_metrics_task',
'schedule': crontab(minute='*/30'),
},
'collect-orderbook-every-5-seconds': {
'task': 'api.tasks.celery_tasks.collect_order_book_task',
'schedule': 5.0, # seconds
'args': ('BTCUSDT',)
},
}
Data Analysis Examples¶
Identifying Market Conditions¶
from api.models import MarketMetric
from datetime import datetime, timedelta
# Get recent metrics
now = datetime.now()
week_ago = now - timedelta(days=7)
btc_dom = MarketMetric.objects.filter(
metric_type='btc_dominance',
timestamp__gte=week_ago
).order_by('-timestamp').first()
fear_greed = MarketMetric.objects.filter(
metric_type='fear_greed_index',
timestamp__gte=week_ago
).order_by('-timestamp').first()
# Analyze conditions
if btc_dom and fear_greed:
btc_value = float(btc_dom.value)
fg_value = float(fear_greed.value)
if fg_value < 25 and btc_value > 50:
print("📊 Market Condition: EXTREME FEAR with BTC dominance rising")
print("💡 Strategy: Consider accumulating quality altcoins")
elif fg_value > 75 and btc_value < 40:
print("📊 Market Condition: EXTREME GREED with low BTC dominance")
print("💡 Strategy: Consider taking profits on altcoins")
Order Book Analysis for Entry/Exit¶
from api.orderbook import OrderBookCollector
from api.exchanges.binance_client import BinanceClient
client = BinanceClient()
collector = OrderBookCollector(client)
# Analyze current order book
data = collector.fetch_and_analyze('BTCUSDT', limit=100)
imbalance = data['imbalance']
support = data['support_resistance']['support']
resistance = data['support_resistance']['resistance']
# Decision logic
if imbalance['pressure'] == 'buying' and imbalance['imbalance_percentage'] > 20:
print("🟢 Strong buying pressure detected")
print(f" First support level: ${support[0]:,.2f}")
print(f" Suggested entry: Around ${support[0]:,.2f}")
elif imbalance['pressure'] == 'selling' and imbalance['imbalance_percentage'] < -20:
print("🔴 Strong selling pressure detected")
print(f" First resistance level: ${resistance[0]:,.2f}")
print(f" Suggested exit: Around ${resistance[0]:,.2f}")
Best Practices¶
Collection Frequency¶
- BTC Dominance & Market Cap: Every 15-60 minutes (API rate limits apply)
- Fear & Greed Index: Once per day (updates daily)
- Order Book: Real-time to every 5 seconds (for active trading)
Storage Considerations¶
- Order book data can grow large quickly. Consider:
- Storing only top 20-50 levels instead of full depth
- Aggregating older data
- Implementing data retention policies
Error Handling¶
All collection functions include error handling and logging. Monitor logs for:
Rate Limiting¶
- CoinGecko: 10-50 requests per minute (free tier)
- Alternative.me: No strict limits but be respectful
- Binance: 1200 requests per minute (weight-based)
Troubleshooting¶
Common Issues¶
Issue: API requests failing
# Check connectivity
curl https://api.coingecko.com/api/v3/ping
curl https://api.alternative.me/fng/
Issue: Database connection errors
Issue: Order book data not updating
- Verify API credentials are correct
- Check exchange client connection:
client.ping() - Ensure exchange client is properly initialized
Debugging¶
Enable debug logging:
Future Enhancements¶
- Add more exchanges for order book data (Coinbase, Kraken, etc.)
- Implement WebSocket streams for real-time order book updates
- Add more market metrics (funding rates, open interest, liquidations)
- Create visualization dashboard
- Implement anomaly detection for market metrics
- Add historical data backfilling
References¶
Support¶
For issues or questions: - Create an issue on GitHub - Check existing documentation in /docs - Review test files for usage examples
License¶
This module is part of the DSTA project and follows the same license.