88 lines
5.0 KiB
Markdown
88 lines
5.0 KiB
Markdown
# DistShap Appendix
|
|
|
|
|
|
**Limitations.**
|
|
At present, DistShap constructs computational graphs on a CPU and distributes them across GPUs to compute Shapley values. Thus, DistShap cannot process graphs that exceed the available CPU memory (258 GB on the system used in our experiments).
|
|
|
|
|
|
**Sampling time and scalability.**
|
|
Although the sampling step generates 30 million subgraphs for 50 nodes, it remains extremely fast due to our strategy of replicating the computation graph across GPUs.
|
|
Figure 9 demonstrates that sampling is highly efficient and scales effectively up to 64 GPUs. A minor slowdown is observed at 128 GPUs, which can be attributed to CUDA kernel overhead.
|
|
|
|
|
|
|
|
*Figure 9: Scalability of sampling on the Reddit dataset. Total sampling time for explaining 50 nodes.*
|
|
|
|
|
|
## GAT Fidelity Results
|
|
|
|
|
|
We evaluate the fidelity of different explanation methods using a 2-layer GAT, with the average Fidelity score computed over 50 test nodes. Bold values indicate the best results. DistShap achieves the best or near-best performance across all settings.
|
|
|
|
|
|
### Fidelity₊ Scores
|
|
|
|
|
|
Table 1 shows Fidelity₊ scores for the top-k most important edges (higher is better).
|
|
|
|
|
|
**Table 1: GAT Fidelity₊ results.**
|
|
|
|
|
|
| Method | Coauthor-CS (10) | Coauthor-CS (30) | Coauthor-Phy (10) | Coauthor-Phy (30) | DBLP (10) | DBLP (30) | ogbn-arxiv (10) | ogbn-arxiv (30) | ogbn-products (10) | ogbn-products (30) | Reddit (10) | Reddit (30) |
|
|
| :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- |
|
|
| Saliency | 0.009 | 0.029 | 0.006 | 0.021 | 0.001 | 0.003 | 0.011 | 0.029 | 0.003 | 0.006 | 0.02 | 0.031 |
|
|
| GNNExplainer | 0.011 | **0.035** | 0.01 | **0.068** | 0.003 | 0.004 | 0.019 | 0.117 | 0.027 | **0.037** | **0.202** | **0.211** |
|
|
| PGExplainer | 0.003 | 0.007 | 0.003 | 0.007 | 0.001 | 0.001 | 0.00 | 0.00 | 0.00 | 0.00 | OOM | OOM |
|
|
| PGMExplainer | 0.005 | 0.01 | 0.004 | 0.006 | 0.001 | 0.001 | 0.009 | 0.009 | 0.002 | 0.003 | OOM | OOM |
|
|
| OrphicX | 0.001 | 0.002 | OOM | OOM | 0.00 | 0.001 | OOM | OOM | OOM | OOM | OOM | OOM |
|
|
| FastDnX | 0.011 | 0.031 | 0.009 | 0.026 | 0.002 | 0.005 | 0.02 | 0.076 | OOM | OOM | OOM | OOM |
|
|
| GNNShap | **0.022** | 0.027 | **0.013** | 0.022 | **0.004** | **0.006** | OOM | OOM | OOM | OOM | OOM | OOM |
|
|
| DistShap | **0.022** | 0.027 | **0.013** | 0.022 | **0.004** | **0.006** | **0.066** | **0.137** | **0.031** | 0.033 | 0.085 | 0.186 |
|
|
|
|
|
|
### Fidelity₋ Scores
|
|
|
|
|
|
Table 2 shows Fidelity₋ scores across different sparsity levels (lower is better), where a sparsity of 0.6 indicates 60% of the least important edges are removed.
|
|
|
|
|
|
**Table 2: GAT Fidelity₋ results.**
|
|
|
|
|
|
| Method | Coauthor-CS (0.3) | Coauthor-CS (0.6) | Coauthor-Phy (0.3) | Coauthor-Phy (0.6) | DBLP (0.3) | DBLP (0.6) | ogbn-arxiv (0.3) | ogbn-arxiv (0.6) | ogbn-products (0.3) | ogbn-products (0.6) | Reddit (0.3) | Reddit (0.6) |
|
|
| :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- |
|
|
| Saliency | 0.018 | 0.033 | 0.012 | 0.024 | 0.009 | 0.015 | 0.034 | 0.058 | 0.034 | 0.055 | 0.049 | 0.106 |
|
|
| GNNExplainer | 0.024 | 0.054 | 0.015 | 0.03 | 0.004 | 0.008 | 0.161 | 0.179 | 0.076 | 0.095 | 0.056 | 0.205 |
|
|
| PGExplainer | 0.049 | 0.129 | 0.04 | 0.11 | 0.005 | 0.018 | 0.032 | 0.049 | 0.101 | 0.135 | OOM | OOM |
|
|
| PGMExplainer | 0.008 | 0.022 | 0.005 | 0.011 | 0.002 | 0.006 | 0.013 | 0.027 | 0.029 | 0.042 | OOM | OOM |
|
|
| OrphicX | 0.014 | 0.038 | OOM | OOM | 0.009 | 0.021 | OOM | OOM | OOM | OOM | OOM | OOM |
|
|
| FastDnX | 0.021 | 0.033 | 0.009 | 0.017 | 0.012 | 0.019 | 0.023 | 0.051 | OOM | OOM | OOM | OOM |
|
|
| GNNShap | **0.004** | **0.012** | **0.002** | 0.006 | **0.001** | **0.003** | OOM | OOM | OOM | OOM | OOM | OOM |
|
|
| DistShap | 0.005 | 0.016 | 0.003 | **0.005** | **0.001** | 0.004 | **0.006** | **0.014** | **0.016** | **0.023** | **0.016** | **0.056** |
|
|
|
|
|
|
## GAT Total Explanation Times
|
|
|
|
|
|
Table 3 presents the total explanation times for 50 nodes. Training times of surrogate models are provided in parentheses. DistShap uses 8 GPUs for Coauthor and DBLP and 128 GPUs for other datasets. (Err: CUDA insufficient resources; OOM: Out Of Memory)
|
|
|
|
|
|
**Table 3: GAT total explanation times for 50 nodes.**
|
|
|
|
|
|
| Method | Coauthor-CS | Coauthor-Phy | DBLP | ogbn-arxiv | ogbn-products | Reddit |
|
|
| :--- | :--- | :--- | :--- | :--- | :--- | :--- |
|
|
| Saliency | 0.68 | 0.92 | 0.57 | 1.33 | 1.36 | 2.25 |
|
|
| GNNExplainer | 23.90 | 28.18 | 20.82 | 360.10 | 466.27 | 57.16 |
|
|
| PGExplainer | 0.30 (874.01) | 0.32 (651.84) | 0.26 (56.76) | 2.70 (424.84) | 4.79 (1858.80) | OOM |
|
|
| PGMExplainer | 237.36 | 407.16 | 223.36 | 7521.50 | 3568.53 | OOM |
|
|
| OrphicX | 52.94 (7706.09) | OOM | 65.68 (1986.32) | OOM | OOM | OOM |
|
|
| FastDnX | 0.11 (66.12) | 0.16 (284.98) | 0.08 (10.18) | 0.60 (21.52) | OOM | OOM |
|
|
| GraphSVX | 663.13 | 1250.90 | 101.83 | OOM | Err | Err |
|
|
| GNNShap | 99.18 | 196.36 | 13.00 | 919.74 | 588.86 | 1139.92 |
|
|
| DistShap | 15.91 | 28.56 | 5.26 | 129.31 | 106.48 | 171.21 |
|
|
|
|
|
|
|