Prometheus TSDB compaction failed

1 개요

Prometheus TSDB compaction failed

2 persist head block: write compaction: add serires

# https://github.com/prometheus/prometheus/issues/3487
# https://github.com/prometheus/prometheus/issues/4108
# https://github.com/prometheus/prometheus/issues/5617
level=error ts=2019-05-31T08:36:52.501Z caller=db.go:363 component=tsdb msg="compaction failed" err="persist head block: write compaction: add series: out-of-order series added with label set ..."

3 reload blocks: head truncate failed

# https://groups.google.com/g/prometheus-users/c/fqRjxSBwXQ4
level=error ts=2020-03-30T23:00:37.520Z caller=db.go:617 component=tsdb msg="compaction failed" err="reload blocks: head truncate failed: create checkpoint: unexpected gap to last checkpoint. expected:3, requested:966" 

# https://www.mail-archive.com/prometheus-users@googlegroups.com/msg05767.html
level=error ts=0000-00-00T00:00:00.000Z caller=db.go:685 component=tsdb msg="compaction failed" err="reload blocks: head truncate failed: create checkpoint: read segments: corruption in segment /var/lib/prometheus2/data/wal/00004615 at 28021780: unexpected full record"

# https://github.com/prometheus/prometheus/issues/6920
level=error ts=2020-03-02T15:00:02.276Z caller=db.go:561 component=tsdb msg="compaction failed" err="reload blocks: head truncate failed: get segment range: segments are not sequential"