Recently, because of the microcode upgrade, some of the workers on Phala will face issues like follows:
phala-pruntime | } phala-pruntime | thread 'main' panicked at 'Initialize Failed: Failed to load checkpoint: Failed to load persistent data phala-pruntime | phala-pruntime | Caused by: phala-pruntime | 0: IoError(Permission denied (os error 13)) phala-pruntime | 1: Permission denied (os error 13)', src/main.rs:161:9 phala-pruntime | note: run with RUST_BACKTRACE=1 environment variable to display a backtrace phala-pruntime | phala-pruntime exited with code 134
Here is a Short-term solution for recovering pRuntime by downgrading intel-microcode:
Please backup the pruntime data folder, and then try on one server first.
- (Optional) Check if Ubuntu upgraded intel-microcode on or around Aug 14, 2023:
cat /var/log/unattended-upgrades/unattended-upgrades.log && gzip -dc /var/log/unattended-upgrades/unattended-upgrades.log.* | grep intel-microcode
If you see something like
2023-08-14 06:57:34,848 INFO Packages that will be upgraded: intel-microcode , and you failed to start your pRuntime after rebooting system, most likely the intel-microcode upgrading is the reason.
- Find the intel-microcode old version you ran before the issue: Index of /ubuntu/pool/main/i/intel-microcode/
Most likely you were running 20230214. If you didn’t reboot your systems for a long time, you may have an older version. You can try to use
last -x to see what was last reboot time before the crash issue.
- Download the version you ran before. Please pick the correct file based on your system version. For example, if 20230214 is what you need, and then:
- If you are running Ubuntu 22.10: intel-microcode_3.20230214.0ubuntu0.22.10.1_amd64.deb
- If you are running Ubuntu 18.04: intel-microcode_3.20230214.0ubuntu0.18.04.1_amd64.deb
- If you are running other Ubuntu versions: intel-microcode_3.20230214.0ubuntu1_amd64.deb
sudo dpkg -i <filename>on the pruntime server.
Reboot your system.
Check if pRuntime runs again.
You can also use
apt-mark hold intel-microcode to avoid auto-upgrading this package.