Skip to content

Instantly share code, notes, and snippets.

@clemente

clemente/gist:221d31f8fd49d95473231b8257cad9c5

Created August 21, 2020 16:48
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save clemente/221d31f8fd49d95473231b8257cad9c5 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save clemente/221d31f8fd49d95473231b8257cad9c5 to your computer and use it in GitHub Desktop.
Download ZIP
elasticsearch errors, OOM
Raw
gistfile1.txt
[2020-08-13T14:58:30,393][WARN ][o.e.m.j.JvmGcMonitorService] [abXmLud] [gc][21145619] overhead, spent [17.2s] collecting in the last [17.3s]
[2020-08-13T14:58:47,743][INFO ][o.e.m.j.JvmGcMonitorService] [abXmLud] [gc][old][21145620][3543761] duration [17.2s], collections [3]/[17.3s], total [17.2s]/[1.8d], memory [3.9gb]->[3.9gb]/[3.9gb], all_pools {[young] [266.2mb]->[266.2mb]/[266.2mb]}{[survivor] [32.2mb]->[33mb]/[33.2mb]}{[old] [3.6gb]->[3.6gb]/[3.6gb]}
[2020-08-13T14:58:47,743][WARN ][o.e.m.j.JvmGcMonitorService] [abXmLud] [gc][21145620] overhead, spent [17.2s] collecting in the last [17.3s]
[2020-08-13T14:59:13,020][INFO ][o.e.m.j.JvmGcMonitorService] [abXmLud] [gc][old][21145621][3543765] duration [25.1s], collections [4]/[25.2s], total [25.1s]/[1.8d], memory [3.9gb]->[3.9gb]/[3.9gb], all_pools {[young] [266.2mb]->[266.2mb]/[266.2mb]}{[survivor] [33mb]->[32.2mb]/[33.2mb]}{[old] [3.6gb]->[3.6gb]/[3.6gb]}
[2020-08-13T14:59:13,020][WARN ][o.e.m.j.JvmGcMonitorService] [abXmLud] [gc][21145621] overhead, spent [25.1s] collecting in the last [25.2s]
[2020-08-13T15:00:06,255][INFO ][o.e.m.j.JvmGcMonitorService] [abXmLud] [gc][old][21145622][3543775] duration [53s], collections [10]/[53.2s], total [53s]/[1.8d], memory [3.9gb]->[3.9gb]/[3.9gb], all_pools {[young] [266.2mb]->[266.2mb]/[266.2mb]}{[survivor] [32.2mb]->[32.8mb]/[33.2mb]}{[old] [3.6gb]->[3.6gb]/[3.6gb]}
[2020-08-13T15:00:06,255][WARN ][o.e.m.j.JvmGcMonitorService] [abXmLud] [gc][21145622] overhead, spent [53s] collecting in the last [53.2s]
[2020-08-13T15:00:29,515][WARN ][o.e.m.j.JvmGcMonitorService] [abXmLud] [gc][21145623] overhead, spent [23.1s] collecting in the last [23.2s]
[2020-08-13T15:01:22,678][INFO ][o.e.m.j.JvmGcMonitorService] [abXmLud] [gc][old][21145624][3543790] duration [53s], collections [10]/[4.6s], total [53s]/[1.8d], memory [3.9gb]->[3.9gb]/[3.9gb], all_pools {[young] [266.2mb]->[266.2mb]/[266.2mb]}{[survivor] [32.6mb]->[33.2mb]/[33.2mb]}{[old] [3.6gb]->[3.6gb]/[3.6gb]}
[2020-08-13T15:01:22,678][WARN ][o.e.m.j.JvmGcMonitorService] [abXmLud] [gc][21145624] overhead, spent [53s] collecting in the last [4.6s]
[2020-08-13T15:05:17,779][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [] fatal error in thread [elasticsearch[abXmLud][write][T#1]], exiting
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3332) ~[?:1.8.0_191]
at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124) ~[?:1.8.0_191]
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448) ~[?:1.8.0_191]
at java.lang.StringBuilder.append(StringBuilder.java:136) ~[?:1.8.0_191]
at org.elasticsearch.index.translog.Checkpoint.write(Checkpoint.java:165) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.index.translog.TranslogWriter.writeCheckpoint(TranslogWriter.java:370) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.index.translog.TranslogWriter.syncUpTo(TranslogWriter.java:328) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.index.translog.Translog.ensureSynced(Translog.java:711) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.index.translog.Translog.ensureSynced(Translog.java:732) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.index.engine.InternalEngine.ensureTranslogSynced(InternalEngine.java:445) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.index.shard.IndexShard$3.write(IndexShard.java:2339) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.common.util.concurrent.AsyncIOProcessor.processList(AsyncIOProcessor.java:107) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.common.util.concurrent.AsyncIOProcessor.drainAndProcess(AsyncIOProcessor.java:99) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.common.util.concurrent.AsyncIOProcessor.put(AsyncIOProcessor.java:82) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.index.shard.IndexShard.sync(IndexShard.java:2361) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.action.support.replication.TransportWriteAction$AsyncAfterWriteAction.run(TransportWriteAction.java:352) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.action.support.replication.TransportWriteAction$WritePrimaryResult.<init>(TransportWriteAction.java:150) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:135) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:112) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:74) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:1018) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryShardReference.perform(TransportReplicationAction.java:996) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.action.support.replication.ReplicationOperation.execute(ReplicationOperation.java:103) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:357) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:297) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:959) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:956) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:270) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:237) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:2213) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryShardReference(TransportReplicationAction.java:968) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction.access$500(TransportReplicationAction.java:98) ~[elasticsearch-6.3.1.jar:6.3.1]
[2020-08-13T15:05:26,491][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [] fatal error in thread [elasticsearch[abXmLud][search][T#3]], exiting
java.lang.OutOfMemoryError: Java heap space
[2020-08-13T15:05:32,069][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [] fatal error in thread [elasticsearch[abXmLud][generic][T#1613]], exiting
java.lang.OutOfMemoryError: Java heap space
[2020-08-13T15:05:32,308][ERROR][o.e.i.e.Engine ] [abXmLud] [filebeat-2020.08.13][4] merge failed
java.lang.OutOfMemoryError: Java heap space
[2020-08-13T15:05:26,486][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [] fatal error in thread [elasticsearch[abXmLud][search][T#6]], exiting
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3181) ~[?:1.8.0_191]
at org.apache.lucene.util.packed.PackedLongValues$Builder.grow(PackedLongValues.java:276) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680 - caomanhdat - 2018-05-09 09:27:24]
at org.apache.lucene.util.packed.DeltaPackedLongValues$Builder.grow(DeltaPackedLongValues.java:95) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680 - caomanhdat - 2018-05-09 09:27:24]
at org.apache.lucene.util.packed.MonotonicLongValues$Builder.grow(MonotonicLongValues.java:96) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680 - caomanhdat - 2018-05-09 09:27:24]
at org.apache.lucene.util.packed.PackedLongValues$Builder.add(PackedLongValues.java:225) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680 - caomanhdat - 2018-05-09 09:27:24]
at org.apache.lucene.index.OrdinalMap.<init>(OrdinalMap.java:251) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680 - caomanhdat - 2018-05-09 09:27:24]
at org.apache.lucene.index.OrdinalMap.build(OrdinalMap.java:168) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680 - caomanhdat - 2018-05-09 09:27:24]
at org.apache.lucene.index.OrdinalMap.build(OrdinalMap.java:147) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680 - caomanhdat - 2018-05-09 09:27:24]
at org.elasticsearch.index.fielddata.ordinals.GlobalOrdinalsBuilder.build(GlobalOrdinalsBuilder.java:65) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.index.fielddata.plain.SortedSetDVOrdinalsIndexFieldData.localGlobalDirect(SortedSetDVOrdinalsIndexFieldData.java:130) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.index.fielddata.plain.SortedSetDVOrdinalsIndexFieldData.localGlobalDirect(SortedSetDVOrdinalsIndexFieldData.java:47) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache.lambda$load$1(IndicesFieldDataCache.java:165) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$$Lambda$2901/351959911.load(Unknown Source) ~[?:?]
at org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:399) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache.load(IndicesFieldDataCache.java:162) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.index.fielddata.plain.SortedSetDVOrdinalsIndexFieldData.loadGlobal(SortedSetDVOrdinalsIndexFieldData.java:118) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.search.aggregations.support.ValuesSource$Bytes$WithOrdinals$FieldData.globalOrdinalsValues(ValuesSource.java:151) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.search.aggregations.support.ValuesSource$Bytes$WithOrdinals.globalMaxOrd(ValuesSource.java:124) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.search.aggregations.bucket.terms.TermsAggregatorFactory.getMaxOrd(TermsAggregatorFactory.java:215) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.search.aggregations.bucket.terms.TermsAggregatorFactory.doCreateInternal(TermsAggregatorFactory.java:137) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.search.aggregations.support.ValuesSourceAggregatorFactory.createInternal(ValuesSourceAggregatorFactory.java:55) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.search.aggregations.AggregatorFactory.create(AggregatorFactory.java:216) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.search.aggregations.AggregatorFactories.createTopLevelAggregators(AggregatorFactories.java:216) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.search.aggregations.AggregationPhase.preProcess(AggregationPhase.java:55) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:105) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$14(IndicesService.java:1134) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.indices.IndicesService$$Lambda$2897/473832838.accept(Unknown Source) ~[?:?]
at org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$15(IndicesService.java:1187) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.indices.IndicesService$$Lambda$2898/1865810227.get(Unknown Source) ~[?:?]
at org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:160) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:143) ~[elasticsearch-6.3.1.jar:6.3.1]
at org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:399) ~[elasticsearch-6.3.1.jar:6.3.1]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment