Vault Owner Misconduct: Disclosure and Solutions

💬 Community
, , ,
#1

On October 7, 2023, several community members reported abnormal declines in Delegation numbers. After checking the on-chain data, we found that multiple Vault Owners were maliciously extracting delegators’ staking rewards by unjustifiably increasing the commission percentage.

The on-chain issue has been resolved through an upgrade, and we will soon provide compensation for all affected delegators within 1 month.

Some Vault holders reported that this conduct was the result of hacking, due to their addresses being stolen. Fortunately, most of the funds have not been lost. These Vault holders will return these funds to the community. This simultaneously serves as a reminder to other community members, to pay attention to the security of accounts and keep your mnemonic phrases safe.

The addresses and Vault PIDs of those caught in the malicious conduct are as follows:

ID owner address PID Cheating date Mark
1 3zz9XwgFmBUh86tEBdWXgoiPmFARSvoDjSgCPkL9r36xAbQ7 4850 2023/10/7 Address Stolen
4880 2023/10/7 Address Stolen
2 41KL4ipqhHovt2U7cXtXQYU2iRnVmuQrotAn2tWzTbYtoxp9 3924 2023/10/6 Address Stolen
3730 2023/10/6 Address Stolen
4343 2023/10/7 Address Stolen
3 42htLs2GGqUavbp4swXUVPnEZvHSRZJVzDvmCRSdFj7Q7tpa 4686 2023/10/6 Address Stolen
4 44qTASZSLp9TxEMazC9GpizhQXvWq17WjMhWfxYyFFkG42jV 4852 2023/10/7 Address Stolen
5 45etpeyNYjT6W9B4i15kUzPteUvVLgFRU5zgbGJNgKmtYsnk 4034 2023/9/8 2023/9/22 2023/9/26 2023/9/30 Unable to get in touch

If you did not hold any delegation in the Vault at the specified time as above, you will not be affected.

Among these, Addresses ID 1-4 are controlled by the same entity, and Address ID 5 is another entity.

The Malicious Vault Owners

We conducted detailed analyses on those engaged in this malicious behaviour, identifying the addresses associated with two culprits:

Malefactor Vault Owner A:

The team associated with these addresses promptly contacted the Phala team recently. They reported the information about the accounts being stolen, and also that most of the misappropriated funds are still within the addresses. This team will cooperate with the solution for this incident and make a full refund to the affected delegators.

**Related Address List: **
42htLs2GGqUavbp4swXUVPnEZvHSRZJVzDvmCRSdFj7Q7tpa
41KL4ipqhHovt2U7cXtXQYU2iRnVmuQrotAn2tWzTbYtoxp9
3zz9XwgFmBUh86tEBdWXgoiPmFARSvoDjSgCPkL9r36xAbQ7
44qTASZSLp9TxEMazC9GpizhQXvWq17WjMhWfxYyFFkG42jV

Public informations registered by the Vault owner:

3zz9XwgFmBUh86tEBdWXgoiPmFARSvoDjSgCPkL9r36xAbQ7
Email: [email protected]
X/Twitter: @unclecat93
Wechat: GeekPhala

41KL4ipqhHovt2U7cXtXQYU2iRnVmuQrotAn2tWzTbYtoxp9
Email: [email protected]
X/Twitter: @yunlianpool
Wechat: yunlianpool

44qTASZSLp9TxEMazC9GpizhQXvWq17WjMhWfxYyFFkG42jV
Email: [email protected]
X/Twitter: @unclecat93

Relationship of Malefactor Vault owner A’s addresses（Link of Map）：

1164×702 70.2 KB

Malefactor Vault Owner B:

Related Address:
45etpeyNYjT6W9B4i15kUzPteUvVLgFRU5zgbGJNgKmtYsnk

Address On-chain Authentication Information:
45etpeyNYjT6W9B4i15kUzPteUvVLgFRU5zgbGJNgKmtYsnk
Email: [email protected]

Dissection of Malicious Logic

The primary intent behind the design of the Vault was to reduce the difficulty Phala users had in staking. Prior to the launch of the Vault, delegators had to understand and manually claim staking rewards every day. The Vault uses an on-chain design similar to that of an LSP, which allows the Vault Owner to use professional management to execute staking strategies on behalf of users at low risk.

A characteristic of the Vault is that only those with high community influence can amass a sufficiently high delegation. The larger the Vault, the more users there are, and the greater the owner’s influence. If Vault owners act maliciously, the larger Vaults could potentially cause significant harm, but the risk of being discovered and losing reputation also increases.

Due to the limitation on the number of transactions in a single block, we are unable to automatically distribute Vault commissions with the distribution of mining rewards. That’s why we adopted a manual rewards withdrawal mechanism for the Vault, requiring Vault owners to manually apply for commission distributions.

There were two key actions in this instance of Vault owners behaving malignantly:

  1. Changing the Commission: The Commission is the percentage of commission set by the Vault owner.
  2. Settling Vault proceeds: When the Vault owner manually chooses to settle the Vault proceeds, the Phala on-chain system would settle the revenues between the Vault Owner and Vault Staker according to the Commission. The settlement reference is the value increment per share brought by all the mining rewards within the Vault between this settlement and the previous one. This increment times the commission ratio, and then times the number of Shares in the Vault, is the number of proceeds for the Vault owners in this settlement.

2362×912 193 KB

Prior to the exposure of this issue, the on-chain implementation allowed the Vault Owner to increase the commission, which not only affected future commissions but also all unsettled commissions from the last settlement until now.

This allowed the Vault Owner to continuously adjust the commission and manipulate settlement times to extract interest profits from the delegators, regardless of the reputation of the Vault.

Loss Calculation

Initial statistics show that 246 users were affected by this malicious conduct, with a total loss of about 704872.53 PHA. Detailed records of the Vault Owners’ cheating profits can be found here: Vault cheater claim records

This month, we will refine the calculation of the actual number of affected individuals and the associated losses through codes, as well as tracing back all records and impacts of similar incidents since Vault function was launched. Becasuse of the complex logic for calculating, it takes time to sort out the information. Thanks for your patience.

Solution

To date, we have forced all Vaults to automatically calculate before modifying the commission through an on-chain upgrade. This mitigates any impact on the earnings of historical delegators during the Vault settlement.

Codes have been upgraded to fix this issue, as initiated by the Phala council in Phala - Motion 79, Khala - Motion 240.

Simultaneously, for all the similar behaviors that we’ve traced back to since the launch of the Vault function, we will compensate for all losses through the mining rewards account and do so via an on-chain referendum. The specific methods of compensation and compensation list will be publicly disclosed in sync with our tracing records.

Also, the primary Vault holder involved in this incident will be refunding the funds associated with this event. The refund records will be publicly displayed in sync with the tracing records.

What Can I Do?

  • If you discover similar behaviors in other Vaults, please feed back the vault ID to this post on the Phala forum as soon as possible: Report bad stakepools&Vaults!. We have a global amb team and they can also help your issue to us. Feel free to contact them when you need help.
  • If you have not staked to any Vaults in the aforementioned list, there’s no need to panic. This incident will not affect you.
  • If you have staked to Vaults in the mentioned list, please stay tuned to our community announcements. We will announce compensation information as soon as the damaged records are completely retraced.
  • Take care of your mnemonic phrases, keep it safe!
3 Likes
Solution for the Vault Owner Misconduct
#2

Special thanks:

This is a belated thank you to the Tucuman team who raised doubts about the Vault reward logic in February. Even though we decided to stick to the original logic of the Vault after multiple discussions at that time, it’s clear that our decision was incorrect.

To rectify this mistake, we categorize this incident as a “High-Level” issue internally, and I will apply for a belated Bug Bounty for the Tucuman team, valued at 4500 USD.

At the same time, we also need to thank community member “High/Stake” for sending the vulnerability report on October 1st, and community member “Suge” for continuous assistance and research after the incident. These efforts have been instrumental in our understanding of the incident. I will submit the proposal to distribute additional bounties to both, each valued at 1800 USD.

2 Likes
#3

Looking forward to that bounty payout, do you know when it will be sent?

#4

Hi team, I understand there is a plan for comensation for ppl who lost funds. Was there anything announced? or is it a plan? Thnak you for your work

#5

That month has now passed.
Can we get an update on the situation please?

Thanks.

1 Like
#6

@doylegxd @Marvin any updates here?

#7

Here is a form of how we calculate the Compensation list

The Sheet “Compensation list” includes the Compensation list and the list of cheating vault owners. For delegators, you can check how much funds can be paid back. And for Vault owners, you will know how much funds you should return.

The Sheet “Cheating address/Elimination list” includes the addresses involved in this issue. Such as the operator of the Cheating vault, the addresses that are used to claim the owner rewards of the cheating vault as well as the addresses that have a close financial turnover connection with the aforementioned two types of addresses.
These three kind of addresses will not be refunded and if you have been targeted, please actively contact the Vault owner to get the fund returned.

The other two sheets are the records of cheating rewards claiming actions, and the influenced delegator list and amount during each action. The summary of this list are the final Compensation list.

Since the payback method changed slightly and we need more on-chain upgrades to launch the refund function. We may take more time to deploy it.
Please be patient and wait for our onchain proposal.

Thanks

4 Likes
#8

Some further explanation：

a. All data in the google doc come from on-chain data, and you can see the “cheater claiming actions record” and “influenced Delegation list” are the raw data our engineer shared with me.
We will share the code how we query it (query all the onchain data of the rewards claim by vault owner). But there are more manually operation actions to calculate the payback list, because not all the claiming actions are cheating so we have to judge it manually. Like how the owner changed the commission, how much time the new commission is changed back, and etc,. That part of work is made manually and that’s the part which really takes time.

b. We can’t move the funds back to the vault owner. The related funds are still owned by the address of vault owner and we can’t force them return it. And we need to compensate the delegators whom suffered from this issue via other funds.
We don’t mind compensating the delegators at the same time as the vault owner paying back to delegators, althrough this would result in double compensation. Just hope that the delegators won’t suffer any loss.
However, we don’t want cheaters to reap more benefits, such as staking to their own pool with other accounts, then collecting part of the rewards as a pool owner, and receiving a sum of money from our compensation. As mentioned above, I tracked the vault owner’s address and tried to do some judgement about the relationship between the delegators and vault owners. The tracking result is shown in the “Cheating address/Elimination list”. It shows the relationship between the pools and the addresses.
For the addresses in “Cheating address/Elimination list”, only they have a directly relationship like “Owner of 1345 also has delegation in 1345”, this kind of result will not be paid back.
The relationship like Owner of 1345 but also have delegations in 4567, he will still get the payback from 4567

Hope the answer above shows why the result takes time. But luckily now we are only waiting for some code for the refund functions, and I am writing the payback proposal as well as the bounty for the contributors in this issue.

1 Like
#9

SQL for how we track how all Vault owners claim rewards:

select
  event.id,
  event.name,
  block.height as block,
  event.index_in_block,
  block.timestamp,
  block.hash,
  (event.args ->> 'pid')::integer as pid,
  event.args ->> 'shares' as shares,
  event.args ->> 'checkoutPrice' as checkout_price
from
  public.event
  join public.block on event.block_id = public.block.id
where
  event.name = 'PhalaVault.OwnerSharesGained'
  and (event.args ->> 'pid')::integer = any ($1)
order by
  block.height asc;
1 Like
#10

Codes for How we calculate the influenced number of Delegation NFTs via Vault owner claim records

import {Parser} from '@json2csv/plainjs'
import csv from 'csvtojson'
import Decimal from 'decimal.js'
import _ from 'lodash'
import {createApi, stakeInfoToShares} from '../phala/lib'
import {log} from '../utils'

Decimal.set({toExpPos: 99, toExpNeg: -99})

const api = await createApi('khala')

const BASE_POOL = '42qnPyfw3sbWMGGtTPPc2YFNZRKPGXswRszyQQjGs2FDxdim'

const json = await csv().fromFile(
  './src/commissionCheat/out/commission_events_mark.csv',
)

const cheatPools = [
  3473, 3481, 3601, 3612, 3617, 3618, 3714, 3730, 3749, 3752, 3796, 3924, 4034,
  4343, 4686, 4720, 4850, 4852, 4880, 4927, 4951,
]

const vaultGainAddress = new Map([
  [3473, '45BsgW5wSLE38P2AeogXRp67wge9pvP1sdm2ZgSWkLEPTBgX'],
  [3481, '45t484rzjQzsDTc6RhbrpkZb3SFvfPZicicjDBH2u4TEcmBs'],
  [3601, '46HgLRQTJ6oGDfiF213BpqRGCaW22zQ564fjDLmcnw8j3Pba'],
  [3612, '41ffwYTKytjq6CvuGZH6956owzm4hQcmi9hx7hoeAAmWFdV4'],
  [3617, '43Wu9TQZ3Dh3WUjtu2uMRtRDMv4ueJzY9p1uu2goL2scUd4h'],
  [3618, '43Wu9TQZ3Dh3WUjtu2uMRtRDMv4ueJzY9p1uu2goL2scUd4h'],
  [3714, '41dFyFd21YnzkjFAWJoMrnShDjfWr46fMxAG9ms8UDtM6JCs'],
  [3730, '41KL4ipqhHovt2U7cXtXQYU2iRnVmuQrotAn2tWzTbYtoxp9'],
  [3749, '42CcPMQbyUsRShqDcEF1ioC8itJQjRcdMZ4QaK9eZgDsuNS2'],
  [3752, '41AVnddVZ9zrmvL5XGyAQjoXA1B4RjwQPNMrwgoMH8cHM9ff'],
  [3796, '42qr283P7XXnNrYCHxZtaVvjx25dfbeyEstwuzFMaQiZ5N8i'],
  [3924, '41KL4ipqhHovt2U7cXtXQYU2iRnVmuQrotAn2tWzTbYtoxp9'],
  [4034, '45etpeyNYjT6W9B4i15kUzPteUvVLgFRU5zgbGJNgKmtYsnk'],
  [4343, '41KL4ipqhHovt2U7cXtXQYU2iRnVmuQrotAn2tWzTbYtoxp9'],
  [4686, '42htLs2GGqUavbp4swXUVPnEZvHSRZJVzDvmCRSdFj7Q7tpa'],
  [4720, '3zxRSK5DquqD1f53e8CXTjqMb9wVBJxPDqb534w77147b5Cz'],
  [4850, '3zz9XwgFmBUh86tEBdWXgoiPmFARSvoDjSgCPkL9r36xAbQ7'],
  [4852, '44qTASZSLp9TxEMazC9GpizhQXvWq17WjMhWfxYyFFkG42jV'],
  [4880, '3zz9XwgFmBUh86tEBdWXgoiPmFARSvoDjSgCPkL9r36xAbQ7'],
  [4927, '45bFuYvSPzT9KBWmkrfkELLH8RYsk5aoPWVpUCjHgU8MrUQh'],
  [4951, '43Wu9TQZ3Dh3WUjtu2uMRtRDMv4ueJzY9p1uu2goL2scUd4h'],
])

const parser = new Parser()

const out = await Promise.all(
  cheatPools.map(async (pid) => {
    const gainAddress = vaultGainAddress.get(pid)
    const events = json.filter(
      (x) =>
        x.pid === pid.toString() && (x.is_cheat === '1'),
    )

    const cid = await api.query.phalaBasePool
      .pools(pid)
      .then((x) => x.unwrap().asVault.basepool.cid.toNumber())

    log(pid, cid)

    const res = []

    for (const event of events) {
      if (event.name !== 'PhalaVault.OwnerSharesGained') continue
      const cheatShares = event.shares
      const {
        block: {
          header: {parentHash},
        },
      } = await api.rpc.chain.getBlock(event.hash)

      const apiAt = await api.at(parentHash)

      const totalShares = await apiAt.query.phalaBasePool
        .pools(pid)
        .then(
          (r) => new Decimal(r.unwrap().asVault.basepool.totalShares.toHex()),
        )
      const withdrawQueue = await apiAt.query.phalaBasePool
        .pools(pid)
        .then((r) => r.unwrap().asVault.basepool.withdrawQueue)

      const nfts = await apiAt.query.rmrkCore.nfts.entries(cid).then((r) =>
        r.map((x) => {
          const owner = x[1].unwrap().owner.asAccountId.toString()
          const nftId = x[0].args[1].toNumber()
          let user: string | undefined = owner
          if (owner === BASE_POOL) {
            const withdrawal = withdrawQueue.find(
              (x) => x.nftId.toNumber() === nftId,
            )
            if (withdrawal != null) {
              user = withdrawal.user.toString()
            }
          }
          return {
            ...event,
            totalShares: totalShares.div(1e12).toString(),
            cid,
            nftId,
            owner,
            user,
            isVaultOwner: user === gainAddress,
          }
        }),
      )

      const nftsInfo = await apiAt.query.rmrkCore.properties
        .multi(nfts.map((x) => [x.cid, x.nftId, 'stake-info']))
        .then((r) =>
          r.map((x) => {
            let shares = new Decimal(0)
            let error = false
            try {
              shares = new Decimal(stakeInfoToShares(x.unwrap()).toString())
            } catch (err) {
              error = true
            }
            const nftShares = shares.div(1e12).toString()
            const nftCheatShares = shares.div(totalShares).times(cheatShares)
            return {
              nftShares,
              nftCheatShares: nftCheatShares.toString(),
              nftCheatValue: nftCheatShares
                .times(event.checkout_price)
                .toDP(12, 0)
                .toString(),
              error,
            }
          }),
        )

      _.merge(nfts, nftsInfo)

      res.push(...nfts.filter((x) => x.nftShares !== '0'))
    }

    return res
  }),
)

await Bun.write(`./src/commissionCheat/out/nfts.csv`, parser.parse(out.flat()))
1 Like
#11

Thanks a lot for those updates.

#12

Hey Community member!

Thank you for your patience and now the solution for the vault issue is finally launched: Solution for the Vault Owner Misconduct

Please convey this message to each other in the community, and visit this page (https://app.phala.network/khala/subsidy) to claim your own subsidy!

Thank you.

2 Likes