Short-term solution for Phala pRuntime stuck issue

💬 Community Miner
,
#1

Recently, because of the microcode upgrade, some of the workers on Phala will face issues like follows:

phala-pruntime  | }
phala-pruntime  | thread 'main' panicked at 'Initialize Failed: Failed to load checkpoint: Failed to load persistent data
phala-pruntime  | 
phala-pruntime  | Caused by:
phala-pruntime  |     0: IoError(Permission denied (os error 13))
phala-pruntime  |     1: Permission denied (os error 13)', src/main.rs:161:9
phala-pruntime  | note: run with RUST_BACKTRACE=1 environment variable to display a backtrace
phala-pruntime  | 
phala-pruntime exited with code 134

Here is a Short-term solution for recovering pRuntime by downgrading intel-microcode:

Please backup the pruntime data folder, and then try on one server first.

  1. (Optional) Check if Ubuntu upgraded intel-microcode on or around Aug 14, 2023:
cat /var/log/unattended-upgrades/unattended-upgrades.log && gzip -dc /var/log/unattended-upgrades/unattended-upgrades.log.* | grep intel-microcode

If you see something like 2023-08-14 06:57:34,848 INFO Packages that will be upgraded: intel-microcode , and you failed to start your pRuntime after rebooting system, most likely the intel-microcode upgrading is the reason.

  1. Find the intel-microcode old version you ran before the issue: Index of /ubuntu/pool/main/i/intel-microcode/

Most likely you were running 20230214. If you didn’t reboot your systems for a long time, you may have an older version. You can try to use last -x to see what was last reboot time before the crash issue.

  1. Download the version you ran before. Please pick the correct file based on your system version. For example, if 20230214 is what you need, and then:
  • If you are running Ubuntu 22.10: intel-microcode_3.20230214.0ubuntu0.22.10.1_amd64.deb
  • If you are running Ubuntu 18.04: intel-microcode_3.20230214.0ubuntu0.18.04.1_amd64.deb
  • If you are running other Ubuntu versions: intel-microcode_3.20230214.0ubuntu1_amd64.deb

  1. Run sudo dpkg -i <filename> on the pruntime server.

  2. Reboot your system.

  3. Check if pRuntime runs again.

You can also use apt-mark hold intel-microcode to avoid auto-upgrading this package.