Prometheus TSDB compaction failed

1 개요

Prometheus TSDB compaction failed

2 persist head block: write compaction: add serires

# https://github.com/prometheus/prometheus/issues/3487
# https://github.com/prometheus/prometheus/issues/4108
# https://github.com/prometheus/prometheus/issues/5617
level=error ts=2019-05-31T08:36:52.501Z caller=db.go:363 component=tsdb msg="compaction failed" err="persist head block: write compaction: add series: out-of-order series added with label set ..."

3 reload blocks: head truncate failed

# https://github.com/prometheus/prometheus/issues/6920
level=error ts=2020-03-02T15:00:02.276Z caller=db.go:561 component=tsdb msg="compaction failed" err="reload blocks: head truncate failed: get segment range: segments are not sequential"