🔥PRB Error Message Summary

💬 Community PRB
#1

I encountered some errors during the operation of the PRB therefore consulted the developers a few days ago. Here are some constructive feedbacks they provided.

I will share a little trick for everyone.

Use crontab to restart the PRB every hour, which is obviously helpful to improve the stability. Some of the less serious problems in it will also be improved after the restart.

Refer to the crontab writing method (the path is modified according to the actual situation)
0 */1 * * * cd /opt/deploy && /usr/local/bin/docker-compose stop trade redis-q lifecycle && /usr/local/bin/docker-compose up -d trade && /usr/local/bin/docker-compose up -d lifecycle

RequestError - ECONNREFUSED

1632722217276 - {"name":"RequestError","code":"ECONNREFUSED","timings":{"start":1632722217273,"socket":1632722217273,"lookup":1632722217274,"error":1632722217275,"phases":{"wait":0,"dns":1,"total":2}},"message":"connect ECONNREFUSED 10.100.135.50:8000","stack":"RequestError: connect ECONNREFUSED 10.100.135.50:8000\n at ClientRequest.<anonymous> (/opt/app/node_modules/.pnpm/[email protected]/node_modules/got/dist/source/core/index.js:956:111)\n at Object.onceWrapper (events.js:520:26)\n at ClientRequest.emit (events.js:412:35)\n at ClientRequest.origin.emit (/opt/app/node_modules/.pnpm/@[email protected]/node_modules/@szmarczak/http-timer/dist/source/index.js:43:20)\n at Socket.socketErrorListener (_http_client.js:475:9)\n at Socket.emit (events.js:412:35)\n at emitErrorNT (internal/streams/destroy.js:106:8)\n at emitErrorCloseNT (internal/streams/destroy.js:74:3)\n at processTicksAndRejections (internal/process/task_queues.js:82:21)\n at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1148:16)"}
  • Context:
  • Reason: pRuntime did not start correctly
  • Solution:

TimeoutError - ETIMEDOUT

1632722342323 - {"name":"TimeoutError","code":"ETIMEDOUT","timings":{"start":1632722312315,"socket":1632722312315,"lookup":1632722312351,"error":1632722342317,"phases":{"wait":0,"dns":36,"total":30002}},"event":"request","message":"Timeout awaiting 'request' for 30000ms","stack":"RequestError: Timeout awaiting 'request' for 30000ms\n at ClientRequest.<anonymous> (/opt/app/node_modules/.pnpm/[email protected]/node_modules/got/dist/source/core/index.js:956:65)\n at Object.onceWrapper (events.js:520:26)\n at ClientRequest.emit (events.js:412:35)\n at ClientRequest.origin.emit (/opt/app/node_modules/.pnpm/@[email protected]/node_modules/@szmarczak/http-timer/dist/source/index.js:43:20)\n at Socket.socketErrorListener (_http_client.js:475:9)\n at Socket.emit (events.js:412:35)\n at emitErrorNT (internal/streams/destroy.js:106:8)\n at emitErrorCloseNT (internal/streams/destroy.js:74:3)\n at processTicksAndRejections (internal/process/task_queues.js:82:21)\n at Timeout.timeoutHandler [as _onTimeout] (/opt/app/node_modules/.pnpm/[email protected]/node_modules/got/dist/source/core/utils/timed-out.js:36:25)\n at listOnTimeout (internal/timers.js:559:11)\n at processTimers (internal/timers.js:500:7)"}
  • Context：
  • Reason：TCP Overtime
  • Soulution：Increase the network throughput of the operating environment

RequestError - ENOTFOUND

1632721028511 - {"name":"RequestError","code":"ENOTFOUND","timings":{"start":1632721028508,"socket":1632721028508,"lookup":1632721028510,"error":1632721028510,"phases":{"wait":0,"dns":2,"total":2}},"message":"getaddrinfo ENOTFOUND node-3-8.cd-farm.phala.network","stack":"RequestError: getaddrinfo ENOTFOUND node-3-8.cd-farm.phala.network\n at ClientRequest.<anonymous> (/opt/app/node_modules/.pnpm/[email protected]/node_modules/got/dist/source/core/index.js:956:111)\n at Object.onceWrapper (events.js:520:26)\n at ClientRequest.emit (events.js:412:35)\n at ClientRequest.origin.emit (/opt/app/node_modules/.pnpm/@[email protected]/node_modules/@szmarczak/http-timer/dist/source/index.js:43:20)\n at Socket.socketErrorListener (_http_client.js:475:9)\n at Socket.emit (events.js:412:35)\n at emitErrorNT (internal/streams/destroy.js:106:8)\n at emitErrorCloseNT (internal/streams/destroy.js:74:3)\n at processTicksAndRejections (internal/process/task_queues.js:82:21)\n at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:69:26)"}
  • Context：
  • Reason：DNS record not found
  • Solution：

SyntaxError

Copy

1632724047731 - {"name":"SyntaxError","message":"Unexpected token I in JSON at position 0","stack":"SyntaxError: Unexpected token I in JSON at position 0\n at JSON.parse (<anonymous>)\n at file:///opt/app/src/lifecycle/pruntime.js:22:24\n at runMicrotasks (<anonymous>)\n at processTicksAndRejections (internal/process/task_queues.js:95:5)\n at async headerSync (file:///opt/app/src/lifecycle/pruntime.js:261:9)\n at async Promise.all (index 0)"}
  • Context：
  • Reason：The data received by pRuntime is incomplete or the data received by prb is incomplete
  • Solution：Increase the network throughput of the operating environment

Error

1632727207074 - {"name":"Error","message":"","stack":"Error\n at Queue._onMessage (/opt/app/node_modules/.pnpm/[email protected]/node_modules/bee-queue/lib/queue.js:138:22)\n at RedisClient.emit (events.js:400:28)\n at return_pub_sub (/opt/app/node_modules/.pnpm/[email protected]/node_modules/redis/index.js:786:18)\n at RedisClient.return_reply (/opt/app/node_modules/.pnpm/[email protected]/node_modules/redis/index.js:833:9)\n at JavascriptRedisParser.returnReply (/opt/app/node_modules/.pnpm/[email protected]/node_modules/redis/index.js:192:18)\n at JavascriptRedisParser.execute (/opt/app/node_modules/.pnpm/[email protected]/node_modules/redis-parser/lib/parser.js:574:12)\n at Socket.<anonymous> (/opt/app/node_modules/.pnpm/[email protected]/node_modules/redis/index.js:274:27)\n at Socket.emit (events.js:400:28)\n at addChunk (internal/streams/readable.js:290:12)\n at readableAddChunk (internal/streams/readable.js:265:9)"}
  • Context：Invalid
  • Reason：Trade queue/chain stuck
  • Solution：

Error

1632778296075 - {"name":"TypeError","message":"Cannot destructure property 'genesisState' of 'appContext.genesis' as it is null.","stack":"TypeError: Cannot destructure property 'genesisState' of 'appContext.genesis' as it is null.\n at initRuntime (file:///opt/app/src/lifecycle/pruntime.js:107:13)\n at runMicrotasks (<anonymous>)\n at processTicksAndRejections (internal/process/task_queues.js:95:5)\n at async run (file:///opt/app/node_modules/.pnpm/[email protected]/node_modules/p-queue/dist/index.js:267:36)"}
  • Context：Can’t get the status of the Worker
  • Reason：
  • Solution：

HTTPError

1632943741161 - {"name":"HTTPError","timings":{"start":1632943741075,"socket":1632943741075,"lookup":1632943741075,"connect":1632943741076,"upload":1632943741076,"response":1632943741160,"end":1632943741160,"phases":{"wait":0,"dns":0,"tcp":1,"request":0,"firstByte":84,"download":0,"total":85}},"message":"Response code 500 (Internal Server Error)","stack":"HTTPError: Response code 500 (Internal Server Error)\n at Request.<anonymous> (/opt/app/node_modules/.pnpm/[email protected]/node_modules/got/dist/source/as-promise/index.js:117:42)\n at runMicrotasks (<anonymous>)\n at processTicksAndRejections (internal/process/task_queues.js:95:5)"}
  • Context：Response code 500 (Internal Server Error)
  • Reason：
  • Solution：

Error

1632869302123 - {"name":"Error","message":"{\"type\":\"pool\",\"key\":\"pid\",\"value\":\"426\"}","stack":"Error: {\"type\":\"pool\",\"key\":\"pid\",\"value\":\"426\"}\n at Queue._onMessage (/opt/app/node_modules/.pnpm/[email protected]/node_modules/bee-queue/lib/queue.js:138:22)\n at RedisClient.emit (events.js:400:28)\n at return_pub_sub (/opt/app/node_modules/.pnpm/[email protected]/node_modules/redis/index.js:786:18)\n at RedisClient.return_reply (/opt/app/node_modules/.pnpm/[email protected]/node_modules/redis/index.js:833:9)\n at JavascriptRedisParser.returnReply (/opt/app/node_modules/.pnpm/[email protected]/node_modules/redis/index.js:192:18)\n at JavascriptRedisParser.execute (/opt/app/node_modules/.pnpm/[email protected]/node_modules/redis-parser/lib/parser.js:574:12)\n at Socket.<anonymous> (/opt/app/node_modules/.pnpm/[email protected]/node_modules/redis/index.js:274:27)\n at Socket.emit (events.js:400:28)\n at addChunk (internal/streams/readable.js:290:12)\n at readableAddChunk (internal/streams/readable.js:265:9)"}
  • Context：
  • Reason：Pool configuration was not found when initializing the worker
  • Solution：

BlockNumberMismatch

1633023184928 - {"payload":{"message":"BlockNumberMismatch"},"signature":"ac6ebcd51bff00e11989748c6f9d530fb2695b957385e24a993a753ffd2114212bc2b710ee7f8286ae99766dbac63bd447a02de30cac7a04ebd917623a8c0683","status":"error","isRuntimeReturnedError":true}
  • Context：
  • Reason: not sure
  • Solution：

RequestError - ECONNRESET

1633082458669 - {"name":"RequestError","code":"ECONNRESET","timings":{"start":1633082458641,"socket":1633082458642,"lookup":1633082458642,"connect":1633082458642,"upload":1633082458642,"error":1633082458662,"phases":{"wait":1,"dns":0,"tcp":0,"request":0,"total":21}},"message":"socket hang up","stack":"RequestError: socket hang up\n at ClientRequest.<anonymous> (/opt/app/node_modules/.pnpm/[email protected]/node_modules/got/dist/source/core/index.js:956:111)\n at Object.onceWrapper (events.js:520:26)\n at ClientRequest.emit (events.js:412:35)\n at ClientRequest.origin.emit (/opt/app/node_modules/.pnpm/@[email protected]/node_modules/@szmarczak/http-timer/dist/source/index.js:43:20)\n at Socket.socketOnEnd (_http_client.js:499:9)\n at Socket.emit (events.js:412:35)\n at endReadableNT (internal/streams/readable.js:1317:12)\n at connResetException (internal/errors.js:628:14)\n at Socket.socketOnEnd (_http_client.js:499:23)\n at Socket.emit (events.js:412:35)\n at endReadableNT (internal/streams/readable.js:1317:12)\n at processTicksAndRejections (internal/process/task_queues.js:82:21)"}
  • Context：
  • Reason：
  • Solution：

Runtime not initialized

1633166278409 - {"payload":{"message":"Runtime not initialized"},"signature":null,"status":"error","isRuntimeReturnedError":true}
  • Context：
  • Reason：
  • Solution：

ReadError

1633201458772 - {"name":"ReadError","message":"Connection to leader lost","stack":"ReadError: Connection to leader lost\n at /opt/app/node_modules/.pnpm/[email protected]/node_modules/levelup/lib/levelup.js:178:15\n at Object.callback (/opt/app/node_modules/.pnpm/[email protected]/node_modules/encoding-down/index.js:75:21)\n at Multilevel._clearRequests (/opt/app/node_modules/.pnpm/github.com+Level+multileveldown@cda3b9c102542b5998_38b07bd5f0ca4d0ff5adf5c8640d2b25/node_modules/multileveldown/leveldown.js:141:18)\n at Duplexify.cleanup (/opt/app/node_modules/.pnpm/github.com+Level+multileveldown@cda3b9c102542b5998_38b07bd5f0ca4d0ff5adf5c8640d2b25/node_modules/multileveldown/leveldown.js:92:12)\n at Duplexify.f (/opt/app/node_modules/.pnpm/[email protected]/node_modules/once/once.js:25:25)\n at onclosenexttick (/opt/app/node_modules/.pnpm/[email protected]/node_modules/end-of-stream/index.js:53:73)\n at processTicksAndRejections (internal/process/task_queues.js:77:11)"}
  • Context：
  • Reason：
  • Solution：

ReadError

1633379249445 - {"name":"ReadError","message":"Corruption: block checksum mismatch: stored = 3848075696, computed = 2750527327 in /var/data/0/030450.sst offset 60200843 size 55977304","stack":"ReadError: Corruption: block checksum mismatch: stored = 3848075696, computed = 2750527327 in /var/data/0/030450.sst offset 60200843 size 55977304\n at /opt/app/node_modules/.pnpm/[email protected]/node_modules/levelup/lib/levelup.js:178:15\n at Object.callback (/opt/app/node_modules/.pnpm/[email protected]/node_modules/encoding-down/index.js:75:21)\n at oncallback (/opt/app/node_modules/.pnpm/github.com+Level+multileveldown@cda3b9c102542b5998_38b07bd5f0ca4d0ff5adf5c8640d2b25/node_modules/multileveldown/leveldown.js:120:18)\n at Decoder.<anonymous> (/opt/app/node_modules/.pnpm/github.com+Level+multileveldown@cda3b9c102542b5998_38b07bd5f0ca4d0ff5adf5c8640d2b25/node_modules/multileveldown/leveldown.js:69:9)\n at Decoder.emit (events.js:400:28)\n at addChunk (/opt/app/node_modules/.pnpm/[email protected]/node_modules/readable-stream/lib/_stream_readable.js:298:12)\n at readableAddChunk (/opt/app/node_modules/.pnpm/[email protected]/node_modules/readable-stream/lib/_stream_readable.js:280:11)\n at Decoder.Readable.push (/opt/app/node_modules/.pnpm/[email protected]/node_modules/readable-stream/lib/_stream_readable.js:241:10)\n at Decoder.Transform.push (/opt/app/node_modules/.pnpm/[email protected]/node_modules/readable-stream/lib/_stream_transform.js:139:32)\n at Decoder._push (/opt/app/node_modules/.pnpm/[email protected]/node_modules/length-prefixed-stream/decode.js:29:8)"}
  • Context：
  • Reason：
  • Solution：

ReadError

1633234834196 - {"name":"ReadError","message":"Corruption: block checksum mismatch: stored = 1005526915, computed = 1799250702 in /var/data/0/043035.sst offset 30280322 size 6772575","stack":"ReadError: Corruption: block checksum mismatch: stored = 1005526915, computed = 1799250702 in /var/data/0/043035.sst offset 30280322 size 6772575\n at /opt/app/node_modules/.pnpm/[email protected]/node_modules/levelup/lib/levelup.js:178:15\n at Object.callback (/opt/app/node_modules/.pnpm/[email protected]/node_modules/encoding-down/index.js:75:21)\n at oncallback (/opt/app/node_modules/.pnpm/github.com+Level+multileveldown@cda3b9c102542b5998_38b07bd5f0ca4d0ff5adf5c8640d2b25/node_modules/multileveldown/leveldown.js:120:18)\n at Decoder.<anonymous> (/opt/app/node_modules/.pnpm/github.com+Level+multileveldown@cda3b9c102542b5998_38b07bd5f0ca4d0ff5adf5c8640d2b25/node_modules/multileveldown/leveldown.js:69:9)\n at Decoder.emit (events.js:400:28)\n at addChunk (/opt/app/node_modules/.pnpm/[email protected]/node_modules/readable-stream/lib/_stream_readable.js:298:12)\n at readableAddChunk (/opt/app/node_modules/.pnpm/[email protected]/node_modules/readable-stream/lib/_stream_readable.js:280:11)\n at Decoder.Readable.push (/opt/app/node_modules/.pnpm/[email protected]/node_modules/readable-stream/lib/_stream_readable.js:241:10)\n at Decoder.Transform.push (/opt/app/node_modules/.pnpm/[email protected]/node_modules/readable-stream/lib/_stream_transform.js:139:32)\n at Decoder._push (/opt/app/node_modules/.pnpm/[email protected]/node_modules/length-prefixed-stream/decode.js:29:8)"}
  • Context：
  • Reason：
  • Solution：

null

1633367841740 - null
  • Context：S_ERROR PRuntime information such as synchronized block data
  • Reason：
  • Solution：

invalid ancestry proof

1633379230591 - {"payload":{"message":"invalid ancestry proof"},"signature":"ea1640bac6453fc31275e042ac4ac666cfd72b39d8d90b7aa46f4354dfaaf908eda532986fcde2058577e4430401a2c50ca5a471c1a6b0849c471fa2ef5bbb81","status":"error","isRuntimeReturnedError":true}
  • Context：S_ERROR PRuntime information such as synchronized block data
  • Reason：
  • Solution：

To be continued…

3 Likes