Load Shedding
Load shedding is a flow control mechanism that deliberately drops or discards incoming data when a system is overwhelmed, helping maintain stability and prevent complete system failure. In time-series databases and streaming systems, load shedding acts as a safety valve that sacrifices some data completeness for system availability.
How load shedding works
Load shedding operates by monitoring system resources and performance metrics, then implementing predefined policies to reject incoming data when thresholds are exceeded. This process can be visualized as a protective valve:
The system continuously evaluates metrics such as:
- CPU utilization
- Memory pressure
- Ingestion rate
- Queue depths
- Processing time
Next generation time-series database
QuestDB is an open-source time-series database optimized for market and heavy industry data. Built from scratch in Java and C++, it offers high-throughput ingestion and fast SQL queries with time-series extensions.
Shedding strategies
Probability-based shedding
Random sampling of incoming data points, dropping a percentage based on system load. This approach maintains statistical representation while reducing volume.
Semantic shedding
Intelligent dropping of data based on business rules and priority levels. For example, keeping all critical alerts while sampling routine telemetry.
Time-based shedding
Dropping data points based on their timestamps, such as maintaining higher resolution for recent data while thinning historical data.
Next generation time-series database
QuestDB is an open-source time-series database optimized for market and heavy industry data. Built from scratch in Java and C++, it offers high-throughput ingestion and fast SQL queries with time-series extensions.
Implementation considerations
Monitoring and alerting
Systems implementing load shedding should provide visibility into:
- Shedding activation frequency
- Volume and types of dropped data
- Resource utilization trends
Recovery strategies
After shedding events, systems may need to:
- Backfill dropped data from source systems
- Adjust retention policies
- Recalibrate shedding thresholds
Impact on data consistency
Load shedding can affect:
- Data completeness guarantees
- Temporal join accuracy
- Aggregation results
- Historical analysis
Real-world applications
Financial markets
High-frequency trading systems might shed lower-priority market data during peak volatility while maintaining critical order flow processing.
Industrial IoT
Manufacturing sensors generating high-frequency measurements might implement semantic shedding to preserve critical quality control data while sampling routine operational metrics.
Infrastructure monitoring
Cloud monitoring systems often use probability-based shedding during large-scale incidents to maintain system responsiveness while preserving statistically significant sampling of events.
The key to successful load shedding is balancing data completeness with system stability, implementing intelligent dropping strategies that preserve the most valuable data while maintaining overall system health.