Compacted Topic

How compaction works in practice

Compacted topics use a key-based compaction strategy where older messages with the same key are eligible for removal, keeping only the newest value. This process, often called log compaction, runs periodically in the background.

The compaction process preserves the temporal ordering of messages while reducing storage requirements. This is particularly important for time-series systems that need to maintain state without keeping every historical update.

Key characteristics and benefits

Efficiency through selective retention

• Maintains data integrity by keeping the latest state
• Reduces storage requirements without losing current values
• Preserves message order within each key's history

Use cases in time-series systems

• State stores for streaming applications
• Configuration management
• Latest-value caches for real-time analytics

The mechanism is particularly valuable when combined with append-only storage systems, as it provides a way to manage growth while maintaining data integrity.

Implementation considerations

Compaction policies

• Time-based retention windows
• Size-based triggers
• Key-based retention rules
• Tombstone record handling

Performance implications

• Background compaction overhead
• Read/write performance during compaction
• Storage optimization benefits

Consider this example of how compaction affects a time-series stream:

The process ensures that while historical changes may be compacted, the system always maintains an accurate current state for each key.

Best practices for compacted topics

1. Key design considerations

   • Choose meaningful keys that support compaction goals
   • Consider key cardinality impact
   • Plan for future scaling needs

2. Monitoring and maintenance

   • Track compaction metrics
   • Monitor storage utilization
   • Validate compaction effectiveness

3. Integration patterns

   • Coordinate with real-time analytics
   • Support event sourcing patterns
   • Enable efficient state recovery

The effectiveness of compacted topics often depends on careful consideration of these factors along with specific use case requirements.

Conclusion

Compacted topics provide a powerful mechanism for managing time-series data growth while maintaining system functionality. By understanding and properly implementing compaction strategies, organizations can optimize their data storage while ensuring access to current state information. This approach is particularly valuable in scenarios requiring efficient state management alongside historical data processing capabilities.