https://pastein.ru/t/_s0
Загрузка данных
PRISM v5.21.0 [9661e39] [sched=+scx_mlfq,scx_cake,scx_pandemonium]
captured: 20260909-024704
SCHEDULER STABILITY
no ejection -- scheduler ran clean
CLEAN-ROOM
state NOISY -- load 1.1/16cpu, uptime 5.4h
SYSTEM
cpu AMD Ryzen 9 5950X 16-Core Processor (16c/16t)
memory 31,2 GiB
gpu amdgpu (1002:747E)
kernel 7.2.3-1-cachyos
scheduler EEVDF
STALL SUSCEPTIBILITY
kernel 7.2.3-1-cachyos
tick HZ=1000 NO_HZ_FULL preempt=PREEMPT_DYNAMIC
cpus 16 governor=powersave deepest-idle=C2 (3 states)
cmdline nohz_full=- isolcpus=-
verdict score 4 -> MODERATE
factors HZ=1000 (tight ticks -- narrows window); idle CPUs go tickless (NO_HZ_IDLE behavior; NO_HZ_FULL built but no nohz_full= mask, so unused); 16 CPUs (many idle at once)
COMPARISON (p99 wake2run, lower is better)
fork-thread EEVDF 36035us -> BPF 8059us (4.5x better), ADAPTIVE 7970us (4.5x better)
ipc EEVDF 31us -> BPF 699us (22.5x worse), ADAPTIVE 656us (21.2x worse)
pcpu-burst EEVDF 13005us -> PANDEMONIUM 23304us (1.8x worse)
sojourn-pressure EEVDF 2486us -> PANDEMONIUM 2934us (1.2x worse)
cachyos EEVDF 1621us -> BPF 579us (2.8x better), ADAPTIVE 605us (2.7x better)
POPULATION (montauk cross-run statistics over 4 run(s), per workload)
cachyos-primes eevdf vs pandemonium-adaptive: N=1 cliff +1.00 perm p=0.200 power n=3
cachyos-primes eevdf vs pandemonium-bpf: N=1 cliff +1.00 perm p=0.200 power n=3
cachyos-primes pandemonium-adaptive vs pandemonium-bpf: N=4 cliff +0.00 perm p=1.000 power n>20
cachyos-stress-ng-cpu-cache-mem eevdf vs pandemonium-adaptive: N=1 cliff +1.00 perm p=0.200 power n=3
cachyos-stress-ng-cpu-cache-mem eevdf vs pandemonium-bpf: N=1 cliff +1.00 perm p=0.200 power n=3
cachyos-stress-ng-cpu-cache-mem pandemonium-adaptive vs pandemonium-bpf: N=4 cliff +0.75 perm p=0.143 power n=11
cachyos-xz-compression eevdf vs pandemonium-adaptive: N=1 cliff +1.00 perm p=0.200 power n=3
cachyos-xz-compression eevdf vs pandemonium-bpf: N=1 cliff +1.00 perm p=0.200 power n=3
cachyos-xz-compression pandemonium-adaptive vs pandemonium-bpf: N=4 cliff +0.00 perm p=0.943 power n>20
cold-wake-16c eevdf vs pandemonium: N=1 cliff +0.00 perm p=1.000 power n=10
cold-wake-starve-16c eevdf vs pandemonium: N=1 cliff +1.00 perm p=0.333 power n>20
fork-thread eevdf vs pandemonium-adaptive: N=1 cliff +1.00 perm p=0.200 power n=3
fork-thread eevdf vs pandemonium-bpf: N=1 cliff +1.00 perm p=0.200 power n=3
fork-thread pandemonium-adaptive vs pandemonium-bpf: N=4 cliff +0.00 perm p=0.857 power n>20
ipc-16c eevdf vs pandemonium-adaptive: N=1 cliff -1.00 perm p=0.667 power n=8
ipc-16c eevdf vs pandemonium-bpf: N=1 cliff -1.00 perm p=0.667 power n=10
ipc-16c pandemonium-adaptive vs pandemonium-bpf: N=2 cliff +0.00 perm p=1.000 power n>20
ipc-32c pandemonium-adaptive vs pandemonium-bpf: N=2 cliff +1.00 perm p=0.333 power n=3
pcpu-burst-16c eevdf vs pandemonium: N=1 cliff -1.00 perm p=0.333 power n=4
sojourn-pressure-16c eevdf vs pandemonium: N=1 cliff +0.00 perm p=1.000 power n>20
cachyos-primes MCB best (lower p99): pandemonium-adaptive (tied), pandemonium-bpf (tied)
cachyos-stress-ng-cpu-cache-mem MCB best (lower p99): pandemonium-adaptive (tied), pandemonium-bpf (tied)
cachyos-xz-compression MCB best (lower p99): pandemonium-adaptive (tied), pandemonium-bpf (tied)
cold-wake-16c MCB best (lower p99): eevdf (tied), pandemonium (tied)
cold-wake-starve-16c MCB best (lower p99): pandemonium
fork-thread MCB best (lower p99): pandemonium-adaptive (tied), pandemonium-bpf (tied)
ipc-16c MCB best (lower p99): eevdf
ipc-32c MCB best (lower p99): pandemonium-bpf
pcpu-burst-16c MCB best (lower p99): eevdf
sojourn-pressure-16c MCB best (lower p99): eevdf (tied), pandemonium (tied)
N<3 per arm -- montauk reports the effect size but the inference is underpowered; repeat runs accumulate here
STORM (cpu_release flood, powersave -- lower storm% is better)
REAL IPI storm (preempt-kick dominant); storm 68/73 intervals (93.2%); reenq/s p50=123832 peak=129002; kick/s=247090 (preempt 117538) reenq/s=116673
WORKLOAD cachyos-EEVDF-primes
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
hot-cpu 15 l2_miss_share 49.1344 HIGH
long-range-dependent migration-rate hurst 0.610936 MED
discord 1056ms profile_vs_mean 2.21137 MED
CAPTURE complete -- 19843 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 14.9k wake2run; p50 3us p99 1615us p999 2528us worst 6900us; 97.8% fast(<100us) / 0.4% mid / 1.7% tick-floor(>=900us); 0.1% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~10926 distinct values; inversion ratio 0.48
WORKLOAD cachyos-EEVDF-stress-ng-cpu-cache-mem
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
long-range-dependent dispatch-rate hurst 0.855107 MED
discord 12200ms profile_vs_mean 2.10708 MED
long-range-dependent migration-rate hurst 0.660625 MED
hot-cpu 15 l2_miss_share 7.21338 LOW
CAPTURE complete -- 164901 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 157k wake2run; p50 2us p99 1058us p999 2248us worst 5616us; 97.3% fast(<100us) / 1.2% mid / 1.5% tick-floor(>=900us); 0.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~100343 distinct values; inversion ratio 0.52
WORKLOAD cachyos-EEVDF-xz-compression
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
long-range-dependent dispatch-rate hurst 0.6834 MED
discord 1089ms profile_vs_mean 1.79375 MED
long-range-dependent migration-rate hurst 0.582772 MED
ioctl-slow 136904/xz duration_ms 0.00069 LOW
hot-cpu 3 l2_miss_share 8.00223 LOW
ioctl-slow 136904/xz duration_ms 0.0005 LOW
CAPTURE complete -- 77212 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 14.7k wake2run; p50 6us p99 1621us p999 2881us worst 3923us; 95.1% fast(<100us) / 1.9% mid / 3.0% tick-floor(>=900us); 0.3% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~13303 distinct values; inversion ratio 0.48
WORKLOAD cachyos-PANDEMONIUM-ADAPTIVE-primes
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
hot-cpu 15 l2_miss_share 47.7957 HIGH
long-range-dependent migration-rate hurst 0.582339 MED
migration-avalanche migration-rate runs 5 MED
CAPTURE complete -- 21179 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 16.2k wake2run; p50 4us p99 538us p999 926us worst 2575us; 91.2% fast(<100us) / 8.7% mid / 0.1% tick-floor(>=900us); 1.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~13187 distinct values; inversion ratio 0.57
WORKLOAD cachyos-PANDEMONIUM-ADAPTIVE-stress-ng-cpu-cache-mem
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
long-range-dependent migration-rate hurst 0.826968 MED
migration-avalanche migration-rate runs 183 MED
discord 1128ms profile_vs_mean 1.7494 MED
long-range-dependent dispatch-rate hurst 0.812187 MED
hot-cpu 15 l2_miss_share 7.08031 LOW
CAPTURE complete -- 169786 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 156k wake2run; p50 4us p99 639us p999 1011us worst 2311us; 93.1% fast(<100us) / 6.5% mid / 0.3% tick-floor(>=900us); 1.8% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~136827 distinct values; inversion ratio 0.50
WORKLOAD cachyos-PANDEMONIUM-ADAPTIVE-xz-compression
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
long-range-dependent dispatch-rate hurst 0.746381 MED
migration-avalanche migration-rate runs 124 MED
discord 74ms profile_vs_mean 2.66276 MED
long-range-dependent migration-rate hurst 0.67573 MED
ioctl-slow 137377/xz duration_ms 0.00051 LOW
hot-cpu 9 l2_miss_share 8.19573 LOW
ioctl-slow 137377/xz duration_ms 0.0004 LOW
CAPTURE complete -- 93391 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 31.2k wake2run; p50 5us p99 605us p999 902us worst 2821us; 92.5% fast(<100us) / 7.3% mid / 0.1% tick-floor(>=900us); 2.4% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~26822 distinct values; inversion ratio 0.52
WORKLOAD cachyos-PANDEMONIUM-BPF-primes
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
hot-cpu 15 l2_miss_share 48.2185 HIGH
long-range-dependent migration-rate hurst 0.615258 MED
migration-avalanche migration-rate runs 9 MED
discord 320ms profile_vs_mean 1.52272 MED
CAPTURE complete -- 21920 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 17.0k wake2run; p50 4us p99 511us p999 714us worst 1000us; 92.2% fast(<100us) / 7.7% mid / 0.0% tick-floor(>=900us); 1.5% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~14054 distinct values; inversion ratio 0.50
WORKLOAD cachyos-PANDEMONIUM-BPF-stress-ng-cpu-cache-mem
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
long-range-dependent dispatch-rate hurst 0.855526 MED
migration-avalanche migration-rate runs 175 MED
discord 753ms profile_vs_mean 1.801 MED
long-range-dependent migration-rate hurst 0.825801 MED
hot-cpu 15 l2_miss_share 7.14229 LOW
CAPTURE complete -- 172300 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 159k wake2run; p50 4us p99 467us p999 883us worst 1002us; 92.2% fast(<100us) / 7.7% mid / 0.1% tick-floor(>=900us); 2.4% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~121691 distinct values; inversion ratio 0.42
WORKLOAD cachyos-PANDEMONIUM-BPF-xz-compression
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
long-range-dependent dispatch-rate hurst 0.73588 MED
migration-avalanche migration-rate runs 112 MED
discord 3438ms profile_vs_mean 1.69677 MED
long-range-dependent migration-rate hurst 0.654091 MED
ioctl-slow 137139/xz duration_ms 0.0007 LOW
hot-cpu 11 l2_miss_share 7.09917 LOW
ioctl-slow 137139/xz duration_ms 0.00039 LOW
CAPTURE complete -- 90635 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 28.4k wake2run; p50 4us p99 579us p999 910us worst 1003us; 92.1% fast(<100us) / 7.7% mid / 0.1% tick-floor(>=900us); 2.9% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~26218 distinct values; inversion ratio 0.48
WORKLOAD cachyos-scx_cake-primes
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
hot-cpu 15 l2_miss_share 50.9557 HIGH
long-range-dependent migration-rate hurst 0.602073 MED
discord 872ms profile_vs_mean 1.61183 MED
CAPTURE complete -- 21458 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 16.5k wake2run; p50 3us p99 1000us p999 2000us worst 2207us; 97.2% fast(<100us) / 0.6% mid / 2.2% tick-floor(>=900us); 0.4% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~9287 distinct values; inversion ratio 0.47
WORKLOAD cachyos-scx_cake-stress-ng-cpu-cache-mem
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
long-range-dependent dispatch-rate hurst 0.845676 MED
migration-avalanche migration-rate runs 50 MED
long-range-dependent migration-rate hurst 0.776894 MED
hot-cpu 15 l2_miss_share 7.10063 LOW
CAPTURE complete -- 176612 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 158k wake2run; p50 3us p99 1000us p999 2000us worst 3000us; 97.4% fast(<100us) / 0.6% mid / 2.0% tick-floor(>=900us); 0.7% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~116206 distinct values; inversion ratio 0.46
WORKLOAD cachyos-scx_cake-xz-compression
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
long-range-dependent dispatch-rate hurst 0.67152 MED
discord 3603ms profile_vs_mean 1.56626 MED
long-range-dependent migration-rate hurst 0.629089 MED
ioctl-slow 137860/xz duration_ms 0.00053 LOW
hot-cpu 14 l2_miss_share 7.34757 LOW
ioctl-slow 137860/xz duration_ms 0.00039 LOW
CAPTURE complete -- 91881 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 29.7k wake2run; p50 4us p99 1000us p999 2000us worst 4244us; 97.0% fast(<100us) / 0.9% mid / 2.1% tick-floor(>=900us); 0.5% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~24602 distinct values; inversion ratio 0.48
WORKLOAD cachyos-scx_mlfq-primes
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
hot-cpu 15 l2_miss_share 58.3531 HIGH
kthread-strand comm#dbb4491c max_strand_ms 16.0038 MED
long-range-dependent dispatch-rate hurst 0.680555 MED
migration-avalanche migration-rate runs 39 MED
discord 670ms profile_vs_mean 1.60973 MED
kthread-strand comm#dca2451f max_strand_ms 9.98579 MED
kthread-strand comm#0878f080 max_strand_ms 8.99068 MED
kthread-strand comm#70339bdc max_strand_ms 7.00426 MED
long-range-dependent migration-rate hurst 0.627235 MED
CAPTURE complete -- 9718 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 4.7k wake2run; p50 10us p99 4997us p999 13998us worst 22006us; 84.3% fast(<100us) / 9.9% mid / 5.8% tick-floor(>=900us); 10.2% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~4661 distinct values; inversion ratio 0.48
REPORT kstrand
VERDICT: 5 strands across 4 per-CPU kthreads; worst HELD strand 16.0ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#dbb4491c 11 1 16.0 16.0 1 0 tid=0 0%
comm#dca2451f 9 1 10.0 10.0 1 0 tid=0 0%
comm#0878f080 0 2 9.0 9.0 2 0 tid=0 0%
comm#70339bdc 4 1 7.0 7.0 1 0 tid=0 0%
WORKLOAD cachyos-scx_mlfq-stress-ng-cpu-cache-mem
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
kthread-strand comm#fa4c32b5 max_strand_ms 9.00604 MED
long-range-dependent dispatch-rate hurst 0.879243 MED
migration-avalanche migration-rate runs 517 MED
kthread-strand comm#180cd039 max_strand_ms 9.00264 MED
kthread-strand comm#0878f080 max_strand_ms 8.00229 MED
kthread-strand comm#0357d774 max_strand_ms 7.00147 MED
long-range-dependent migration-rate hurst 0.855207 MED
hot-cpu 15 l2_miss_share 7.52642 LOW
CAPTURE complete -- 58334 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 48.3k wake2run; p50 9us p99 4949us p999 12999us worst 29721us; 88.1% fast(<100us) / 6.8% mid / 5.1% tick-floor(>=900us); 9.8% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~44534 distinct values; inversion ratio 0.54
REPORT kstrand
VERDICT: 12 strands across 4 per-CPU kthreads; worst HELD strand 9.0ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#fa4c32b5 3 4 9.0 9.0 4 0 tid=0 0%
comm#180cd039 5 1 9.0 9.0 1 0 tid=0 0%
comm#0878f080 0 6 8.0 8.0 6 0 tid=0 0%
comm#0357d774 1 1 7.0 7.0 1 0 tid=0 0%
WORKLOAD cachyos-scx_mlfq-xz-compression
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
kthread-strand comm#d1030141 max_strand_ms 17.0282 MED
long-range-dependent dispatch-rate hurst 0.768218 MED
migration-avalanche migration-rate runs 138 MED
discord 4034ms profile_vs_mean 1.58652 MED
kthread-strand comm#cfa27126 max_strand_ms 17.0035 MED
kthread-strand comm#ca814de8 max_strand_ms 7.2744 MED
kthread-strand comm#4f7aa3e7 max_strand_ms 7.0087 MED
kthread-strand comm#df364860 max_strand_ms 7.00155 MED
kthread-strand comm#a3cc20ce max_strand_ms 6.00498 MED
kthread-strand comm#58bc7c1b max_strand_ms 6.00186 MED
kthread-strand comm#0878f080 max_strand_ms 5.69456 MED
long-range-dependent migration-rate hurst 0.665798 MED
ioctl-slow 137620/xz duration_ms 0.00049 LOW
hot-cpu 8 l2_miss_share 7.5908 LOW
ioctl-slow 137620/xz duration_ms 0.00038 LOW
CAPTURE complete -- 71125 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 8.9k wake2run; p50 12us p99 4994us p999 14656us worst 20001us; 84.5% fast(<100us) / 8.8% mid / 6.6% tick-floor(>=900us); 12.7% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~8906 distinct values; inversion ratio 0.41
LOCATED: 1 structured region(s) NEARLY_SORTED[29%..34%]
REPORT kstrand
VERDICT: 9 strands across 8 per-CPU kthreads; worst HELD strand 17.0ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#d1030141 15 1 17.0 17.0 1 0 tid=0 0%
comm#cfa27126 15 2 17.0 17.0 2 0 tid=0 0%
comm#ca814de8 14 1 7.3 7.3 1 0 tid=0 0%
comm#4f7aa3e7 3 1 7.0 7.0 1 0 tid=0 0%
comm#df364860 10 1 7.0 7.0 1 0 tid=0 0%
comm#a3cc20ce 13 1 6.0 6.0 1 0 tid=0 0%
comm#58bc7c1b 7 1 6.0 6.0 1 0 tid=0 0%
comm#0878f080 0 1 5.7 5.7 1 0 tid=0 0%
WORKLOAD cold-wake-16c-EEVDF
THERMAL/POWER
cpu temp peak 50.6 C avg 47.6 C
power avg 70.1 W peak 98.0 W
idle C2 avg 95.7% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
hot-cpu 1 l2_miss_share 84.165 HIGH
CAPTURE complete -- 840 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 47 wake2run; p50 3us p99 11us p999 11us worst 11us; 100.0% fast(<100us) / 0.0% mid / 0.0% tick-floor(>=900us); 0.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~39 distinct values; inversion ratio 0.46
COLD-WAKE: cold-cache worst 4.3x at 32KB (0.03ms cold); CPU freq ramp worst +42%
WORKLOAD cold-wake-16c-PANDEMONIUM
THERMAL/POWER
cpu temp peak 47.6 C avg 47.2 C
power avg 69.7 W peak 98.7 W
idle C2 avg 95.7% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
hot-cpu 1 l2_miss_share 85.6003 HIGH
CAPTURE complete -- 827 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 42 wake2run; p50 4us p99 10us p999 10us worst 10us; 100.0% fast(<100us) / 0.0% mid / 0.0% tick-floor(>=900us); 0.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~40 distinct values; inversion ratio 0.55
COLD-WAKE: cold-cache worst 4.4x at 32KB (0.03ms cold); CPU freq ramp worst +46%
WORKLOAD cold-wake-16c-scx_cake
THERMAL/POWER
cpu temp peak 53.0 C avg 47.6 C
power avg 69.9 W peak 109.9 W
idle C2 avg 95.4% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
hot-cpu 15 l2_miss_share 84.3886 HIGH
CAPTURE complete -- 874 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 44 wake2run; p50 3us p99 9us p999 9us worst 9us; 100.0% fast(<100us) / 0.0% mid / 0.0% tick-floor(>=900us); 0.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~39 distinct values; inversion ratio 0.63
COLD-WAKE: cold-cache worst 4.7x at 32KB (0.03ms cold); CPU freq ramp worst +37%
WORKLOAD cold-wake-16c-scx_mlfq
THERMAL/POWER
cpu temp peak 49.2 C avg 47.1 C
power avg 69.7 W peak 105.0 W
idle C2 avg 95.3% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
hot-cpu 1 l2_miss_share 41.668 HIGH
long-range-dependent dispatch-rate hurst 0.532965 MED
CAPTURE complete -- 851 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 50 wake2run; p50 5us p99 12us p999 12us worst 12us; 100.0% fast(<100us) / 0.0% mid / 0.0% tick-floor(>=900us); 0.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~47 distinct values; inversion ratio 0.52
COLD-WAKE: cold-cache worst 4.8x at 32KB (0.03ms cold); CPU freq ramp worst +44%
WORKLOAD cold-wake-starve-16c-EEVDF
THERMAL/POWER
cpu temp peak 57.6 C avg 50.6 C
power avg 83.3 W peak 107.1 W
idle C2 avg 94.1% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
discord 28477ms profile_vs_mean 4.70381 HIGH
hot-cpu 15 l2_miss_share 51.5373 HIGH
mid-trace-death comm#e3e5febe/172580 before_end_s 35.0601 MED
long-range-dependent dispatch-rate hurst 0.923207 MED
mid-trace-death comm#e3e5febe/172609 before_end_s 28.0401 MED
mid-trace-death comm#e3e5febe/172702 before_end_s 7.01967 MED
CAPTURE complete -- 212839 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 95.3k wake2run; p50 1us p99 9us p999 46us worst 999us; 100.0% fast(<100us) / 0.0% mid / 0.0% tick-floor(>=900us); 0.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~71459 distinct values; inversion ratio 0.56
WORKLOAD cold-wake-starve-16c-PANDEMONIUM
THERMAL/POWER
cpu temp peak 58.0 C avg 50.8 C
power avg 85.3 W peak 107.0 W
idle C2 avg 93.9% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
discord 52104ms profile_vs_mean 3.74757 HIGH
hot-cpu 15 l2_miss_share 57.8357 HIGH
mid-trace-death comm#e3e5febe/174371 before_end_s 35.024 MED
long-range-dependent dispatch-rate hurst 0.920982 MED
mid-trace-death comm#e3e5febe/174400 before_end_s 28.0029 MED
mid-trace-death comm#e3e5febe/174487 before_end_s 7.00036 MED
CAPTURE complete -- 215208 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 97.6k wake2run; p50 2us p99 6us p999 12us worst 1349us; 100.0% fast(<100us) / 0.0% mid / 0.0% tick-floor(>=900us); 0.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~68605 distinct values; inversion ratio 0.62
LOCATED: 4 structured region(s) FEW_UNIQUE[25%..26%], FEW_UNIQUE[27%..28%], FEW_UNIQUE[29%..30%], FEW_UNIQUE[33%..34%]
WORKLOAD cold-wake-starve-16c-scx_cake
THERMAL/POWER
cpu temp peak 58.5 C avg 50.6 C
power avg 81.6 W peak 105.7 W
idle C2 avg 94.3% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
discord 49055ms profile_vs_mean 3.65122 HIGH
hot-cpu 15 l2_miss_share 63.2235 HIGH
mid-trace-death comm#e3e5febe/173780 before_end_s 35.0436 MED
long-range-dependent dispatch-rate hurst 0.921973 MED
mid-trace-death comm#e3e5febe/173809 before_end_s 28.041 MED
mid-trace-death comm#e3e5febe/173896 before_end_s 7.01973 MED
CAPTURE complete -- 214413 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 97.0k wake2run; p50 2us p99 5us p999 11us worst 1999us; 100.0% fast(<100us) / 0.0% mid / 0.0% tick-floor(>=900us); 0.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~66668 distinct values; inversion ratio 0.64
LOCATED: 3 structured region(s) FEW_UNIQUE[3%..4%], FEW_UNIQUE[33%..33%], FEW_UNIQUE[34%..35%]
WORKLOAD cold-wake-starve-16c-scx_mlfq
THERMAL/POWER
cpu temp peak 58.2 C avg 49.9 C
power avg 83.0 W peak 106.6 W
idle C2 avg 94.1% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
discord 51224ms profile_vs_mean 3.45578 HIGH
hot-cpu 15 l2_miss_share 58.5883 HIGH
mid-trace-death comm#e3e5febe/173177 before_end_s 35.0176 MED
long-range-dependent dispatch-rate hurst 0.915594 MED
mid-trace-death comm#e3e5febe/173206 before_end_s 28.0046 MED
mid-trace-death comm#e3e5febe/173299 before_end_s 7.00293 MED
CAPTURE complete -- 198154 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 85.8k wake2run; p50 1us p99 3948us p999 4948us worst 4955us; 97.9% fast(<100us) / 0.1% mid / 2.0% tick-floor(>=900us); 0.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~54943 distinct values; inversion ratio 0.66
LOCATED: 6 structured region(s) NEARLY_SORTED[4%..4%], FEW_UNIQUE[13%..16%], FEW_UNIQUE[17%..17%], FEW_UNIQUE[18%..20%], FEW_UNIQUE[20%..22%] +1 more
WORKLOAD fork-thread-EEVDF
THERMAL/POWER
cpu temp peak 59.5 C avg 53.3 C
power avg 99.3 W peak 165.4 W
idle C2 avg 75.9% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
order-starved tid=866 passover_picks 13706 HIGH
held-cpu tid=141881 held_ms 10.943 HIGH
self-preempt tid=1637 stint_wake_ratio 8548 HIGH
self-preempt tid=1710 stint_wake_ratio 8505 HIGH
self-preempt tid=1706 stint_wake_ratio 8498 HIGH
self-preempt tid=991 stint_wake_ratio 8485 HIGH
order-starved tid=1711 passover_picks 10743 HIGH
held-cpu tid=141880 held_ms 10.069 HIGH
self-preempt tid=1047 stint_wake_ratio 8451 HIGH
self-preempt tid=1712 stint_wake_ratio 8449 HIGH
self-preempt tid=1713 stint_wake_ratio 8447 HIGH
order-starved tid=1707 passover_picks 8629 HIGH
held-cpu tid=141833 held_ms 9.56137 HIGH
self-preempt tid=1635 stint_wake_ratio 8434 HIGH
hot-waker 141962 wakes_issued 1880 MED
long-range-dependent dispatch-rate hurst 0.887256 MED
migration-avalanche migration-rate runs 146 MED
discord 3452ms profile_vs_mean 1.71151 MED
hot-waker 141803 wakes_issued 1584 MED
hot-waker 141885 wakes_issued 1488 MED
long-range-dependent migration-rate hurst 0.707074 MED
hot-cpu 2 l2_miss_share 6.65754 LOW
CAPTURE complete -- 885573 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 87.9k wake2run; p50 109us p99 36035us p999 75771us worst 158394us; 48.7% fast(<100us) / 11.8% mid / 39.5% tick-floor(>=900us); 2.3% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~85174 distinct values; inversion ratio 0.62
REPORT dispatch-stall
VERDICT: 34.7k saturated wakes over the 900us floor; PREEMPT-STARVED 1% (0 intervening picks) / ORDER-STARVED 99% (CPU served others first); avg 54.5 pass-overs, p99 343 pass-overs
WORKLOAD fork-thread-PANDEMONIUM-ADAPTIVE
THERMAL/POWER
cpu temp peak 59.2 C avg 53.4 C
power avg 96.9 W peak 163.3 W
idle C2 avg 77.2% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
order-starved tid=144925 passover_picks 747 HIGH
held-cpu tid=144276 held_ms 6.58115 HIGH
self-preempt tid=144024 stint_wake_ratio 10591 HIGH
self-preempt tid=867 stint_wake_ratio 9164 HIGH
self-preempt tid=1711 stint_wake_ratio 9153 HIGH
self-preempt tid=1707 stint_wake_ratio 9123 HIGH
order-starved tid=144550 passover_picks 691 HIGH
held-cpu tid=144277 held_ms 6.27221 HIGH
self-preempt tid=1710 stint_wake_ratio 9098 HIGH
self-preempt tid=1634 stint_wake_ratio 9080 HIGH
self-preempt tid=1047 stint_wake_ratio 8883 HIGH
order-starved tid=144211 passover_picks 660 HIGH
held-cpu tid=144186 held_ms 3.82753 HIGH
self-preempt tid=1636 stint_wake_ratio 8862 HIGH
hot-waker 144239 wakes_issued 1973 MED
long-range-dependent dispatch-rate hurst 0.810272 MED
migration-avalanche migration-rate runs 1330 MED
discord 5639ms profile_vs_mean 1.70558 MED
hot-waker 144276 wakes_issued 1942 MED
hot-waker 144148 wakes_issued 1905 MED
long-range-dependent migration-rate hurst 0.800133 MED
hot-cpu 15 l2_miss_share 8.04571 LOW
CAPTURE complete -- 2134148 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 218k wake2run; p50 137us p99 7970us p999 22405us worst 41818us; 46.1% fast(<100us) / 31.3% mid / 22.6% tick-floor(>=900us); 4.9% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~218120 distinct values; inversion ratio 0.64
LOCATED: 7 structured region(s) NEARLY_SORTED[0%..0%], NEARLY_SORTED[2%..3%], NEARLY_SORTED[16%..16%], NEARLY_SORTED[58%..58%], NEARLY_SORTED[65%..66%] +2 more
REPORT dispatch-stall
VERDICT: 49.3k saturated wakes over the 900us floor; PREEMPT-STARVED 0% (0 intervening picks) / ORDER-STARVED 100% (CPU served others first); avg 56.2 pass-overs, p99 255 pass-overs
WORKLOAD fork-thread-PANDEMONIUM-BPF
THERMAL/POWER
cpu temp peak 59.2 C avg 52.2 C
power avg 92.5 W peak 154.4 W
idle C2 avg 80.7% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
order-starved tid=142964 passover_picks 554 HIGH
held-cpu tid=143012 held_ms 4.8412 HIGH
self-preempt tid=142853 stint_wake_ratio 8939 HIGH
self-preempt tid=867 stint_wake_ratio 8794 HIGH
self-preempt tid=1707 stint_wake_ratio 8774 HIGH
self-preempt tid=1713 stint_wake_ratio 8755 HIGH
order-starved tid=143045 passover_picks 500 HIGH
held-cpu tid=143102 held_ms 4.32151 HIGH
self-preempt tid=1635 stint_wake_ratio 8744 HIGH
self-preempt tid=1711 stint_wake_ratio 8716 HIGH
self-preempt tid=1047 stint_wake_ratio 8704 HIGH
order-starved tid=143805 passover_picks 499 HIGH
held-cpu tid=143151 held_ms 3.78196 HIGH
self-preempt tid=1637 stint_wake_ratio 8700 HIGH
long-range-dependent dispatch-rate hurst 0.835614 MED
migration-avalanche migration-rate runs 1246 MED
discord 5932ms profile_vs_mean 1.69964 MED
long-range-dependent migration-rate hurst 0.773154 MED
hot-cpu 8 l2_miss_share 7.11278 LOW
CAPTURE complete -- 1969458 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 215k wake2run; p50 117us p99 8059us p999 12688us worst 41732us; 48.1% fast(<100us) / 28.9% mid / 23.0% tick-floor(>=900us); 4.9% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~215315 distinct values; inversion ratio 0.53
LOCATED: 2 structured region(s) NEARLY_SORTED[0%..0%], NEARLY_SORTED[12%..12%]
REPORT dispatch-stall
VERDICT: 49.4k saturated wakes over the 900us floor; PREEMPT-STARVED 0% (0 intervening picks) / ORDER-STARVED 100% (CPU served others first); avg 48.7 pass-overs, p99 193 pass-overs
WORKLOAD ipc-EEVDF-16c
THERMAL/POWER
cpu temp peak 55.2 C avg 50.1 C
power avg 102.9 W peak 142.6 W
idle C2 avg 54.2% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
held-cpu comm#8a5f93b2 held_ms 193.896 HIGH
hot-waker 162942 wakes_issued 45188 HIGH
self-preempt tid=162889 stint_wake_ratio 26610 HIGH
hot-cpu 8 l2_miss_share 18.8161 HIGH
self-preempt tid=162890 stint_wake_ratio 21477 HIGH
self-preempt tid=162877 stint_wake_ratio 20557 HIGH
self-preempt tid=162879 stint_wake_ratio 17715 HIGH
held-cpu comm#8a5f93b2 held_ms 129.981 HIGH
self-preempt tid=162878 stint_wake_ratio 17624 HIGH
self-preempt tid=162870 stint_wake_ratio 16114 HIGH
self-preempt tid=162885 stint_wake_ratio 14847 HIGH
held-cpu comm#8a5f93b2 held_ms 121.56 HIGH
self-preempt tid=162873 stint_wake_ratio 14630 HIGH
order-starved comm#8a5f93b2 passover_picks 28 MED
long-range-dependent dispatch-rate hurst 0.849071 MED
discord 8158ms profile_vs_mean 1.98916 MED
order-starved comm#328ab0c6 passover_picks 17 MED
order-starved comm#8a5f93b2 passover_picks 16 MED
long-range-dependent migration-rate hurst 0.55421 MED
recvfrom-slow 162915/pand-ipc duration_ms 1.70308 LOW
recvfrom-slow 162917/pand-ipc duration_ms 1.69973 LOW
recvfrom-slow 162915/pand-ipc duration_ms 1.69558 LOW
recvfrom-slow 162917/pand-ipc duration_ms 1.69292 LOW
recvfrom-slow 162915/pand-ipc duration_ms 1.68871 LOW
ioctl-slow 162935/python3 duration_ms 0.00098 LOW
ioctl-slow 162935/python3 duration_ms 0.00048 LOW
ioctl-slow 162935/python3 duration_ms 0.00046 LOW
ioctl-slow 162935/python3 duration_ms 0.00045 LOW
ioctl-slow 162935/python3 duration_ms 0.00039 LOW
CAPTURE complete -- 1837960 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 225k wake2run; p50 1us p99 31us p999 1504us worst 3270us; 99.5% fast(<100us) / 0.2% mid / 0.3% tick-floor(>=900us); 0.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~95079 distinct values; inversion ratio 0.35
LOCATED: 12 structured region(s) FEW_UNIQUE[0%..1%], FEW_UNIQUE[1%..2%], FEW_UNIQUE[2%..2%], FEW_UNIQUE[3%..10%], FEW_UNIQUE[11%..15%] +7 more
REPORT dispatch-stall
VERDICT: 608 saturated wakes over the 900us floor; PREEMPT-STARVED 80% (0 intervening picks) / ORDER-STARVED 20% (CPU served others first); avg 1.4 pass-overs, p99 3 pass-overs
WORKLOAD ipc-PANDEMONIUM-ADAPTIVE-16c
THERMAL/POWER
cpu temp peak 55.0 C avg 50.8 C
power avg 119.6 W peak 142.3 W
idle C2 avg 32.7% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
order-starved comm#8a5f93b2 passover_picks 198 HIGH
held-cpu comm#c233d180 held_ms 4.13838 HIGH
self-preempt tid=163424 stint_wake_ratio 37108 HIGH
self-preempt tid=163420 stint_wake_ratio 36108 HIGH
self-preempt tid=163421 stint_wake_ratio 29938 HIGH
self-preempt tid=163422 stint_wake_ratio 29036 HIGH
order-starved comm#279cf575 passover_picks 72 HIGH
held-cpu comm#8a5f93b2 held_ms 2.97799 HIGH
self-preempt tid=163426 stint_wake_ratio 22691 HIGH
self-preempt tid=163433 stint_wake_ratio 20423 HIGH
self-preempt tid=163423 stint_wake_ratio 13042 HIGH
order-starved comm#279cf575 passover_picks 63 HIGH
held-cpu comm#8a5f93b2 held_ms 1.00131 HIGH
self-preempt tid=1707 stint_wake_ratio 9407 HIGH
hot-waker 163495 wakes_issued 78748 MED
long-range-dependent dispatch-rate hurst 0.884828 MED
migration-avalanche migration-rate runs 47 MED
discord 2995ms profile_vs_mean 1.86679 MED
hot-cpu 0 l2_miss_share 17.4346 MED
long-range-dependent migration-rate hurst 0.644 MED
recvfrom-slow 163472/pand-ipc duration_ms 2.68268 LOW
recvfrom-slow 163470/pand-ipc duration_ms 2.65885 LOW
recvfrom-slow 163472/pand-ipc duration_ms 2.64931 LOW
recvfrom-slow 163470/pand-ipc duration_ms 2.61605 LOW
recvfrom-slow 163472/pand-ipc duration_ms 1.99744 LOW
ioctl-slow 163488/python3 duration_ms 0.00114 LOW
ioctl-slow 163488/python3 duration_ms 0.00049 LOW
ioctl-slow 163488/python3 duration_ms 0.00047 LOW
ioctl-slow 163488/python3 duration_ms 0.00044 LOW
ioctl-slow 163488/python3 duration_ms 0.00039 LOW
CAPTURE complete -- 1901864 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 245k wake2run; p50 3us p99 656us p999 973us worst 2902us; 98.0% fast(<100us) / 1.8% mid / 0.2% tick-floor(>=900us); 0.1% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~203281 distinct values; inversion ratio 0.54
LOCATED: 7 structured region(s) FEW_UNIQUE[0%..1%], FEW_UNIQUE[56%..57%], FEW_UNIQUE[58%..58%], FEW_UNIQUE[59%..60%], FEW_UNIQUE[60%..60%] +2 more
REPORT dispatch-stall
VERDICT: 464 saturated wakes over the 900us floor; PREEMPT-STARVED 2% (0 intervening picks) / ORDER-STARVED 98% (CPU served others first); avg 1.8 pass-overs, p99 4 pass-overs
WORKLOAD ipc-PANDEMONIUM-BPF-16c
THERMAL/POWER
cpu temp peak 55.8 C avg 51.9 C
power avg 102.4 W peak 142.0 W
idle C2 avg 55.0% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
order-starved comm#8a5f93b2 passover_picks 134 HIGH
held-cpu comm#8a5f93b2 held_ms 1.69534 HIGH
self-preempt tid=163122 stint_wake_ratio 49202 HIGH
self-preempt tid=163127 stint_wake_ratio 34894 HIGH
self-preempt tid=163126 stint_wake_ratio 31057 HIGH
self-preempt tid=163128 stint_wake_ratio 26039 HIGH
order-starved comm#279cf575 passover_picks 44 HIGH
self-preempt tid=163129 stint_wake_ratio 22309 HIGH
self-preempt tid=163143 stint_wake_ratio 19781 HIGH
self-preempt tid=163136 stint_wake_ratio 17152 HIGH
order-starved comm#279cf575 passover_picks 41 HIGH
self-preempt tid=163123 stint_wake_ratio 12259 HIGH
hot-waker 163197 wakes_issued 79223 MED
long-range-dependent dispatch-rate hurst 0.875319 MED
migration-avalanche migration-rate runs 12 MED
discord 5553ms profile_vs_mean 1.74025 MED
hot-cpu 0 l2_miss_share 17.2292 MED
long-range-dependent migration-rate hurst 0.552655 MED
recvfrom-slow 163170/pand-ipc duration_ms 1.5767 LOW
recvfrom-slow 163172/pand-ipc duration_ms 1.56964 LOW
recvfrom-slow 163170/pand-ipc duration_ms 1.03695 LOW
recvfrom-slow 163170/pand-ipc duration_ms 1.03286 LOW
recvfrom-slow 163170/pand-ipc duration_ms 1.03272 LOW
ioctl-slow 163188/python3 duration_ms 0.00088 LOW
ioctl-slow 163188/python3 duration_ms 0.00053 LOW
ioctl-slow 163188/python3 duration_ms 0.00048 LOW
ioctl-slow 163188/python3 duration_ms 0.00047 LOW
ioctl-slow 163188/python3 duration_ms 0.0004 LOW
CAPTURE complete -- 1922782 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 244k wake2run; p50 3us p99 699us p999 920us worst 2795us; 97.9% fast(<100us) / 1.9% mid / 0.1% tick-floor(>=900us); 0.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~209489 distinct values; inversion ratio 0.36
LOCATED: 1 structured region(s) NEARLY_SORTED[17%..17%]
REPORT dispatch-stall
VERDICT: 281 saturated wakes over the 900us floor; PREEMPT-STARVED 0% (0 intervening picks) / ORDER-STARVED 100% (CPU served others first); avg 1.8 pass-overs, p99 4 pass-overs
WORKLOAD ipc-scx_cake-16c
THERMAL/POWER
cpu temp peak 55.1 C avg 52.5 C
power avg 119.5 W peak 140.9 W
idle C2 avg 29.8% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
order-starved comm#328ab0c6 passover_picks 43 HIGH
held-cpu comm#8a5f93b2 held_ms 46.6252 HIGH
self-preempt tid=164026 stint_wake_ratio 17015 HIGH
self-preempt tid=164024 stint_wake_ratio 16652 HIGH
self-preempt tid=164022 stint_wake_ratio 16617 HIGH
self-preempt tid=164032 stint_wake_ratio 16556 HIGH
order-starved comm#328ab0c6 passover_picks 37 HIGH
held-cpu comm#8a5f93b2 held_ms 37.8319 HIGH
self-preempt tid=164021 stint_wake_ratio 16435 HIGH
self-preempt tid=164027 stint_wake_ratio 16427 HIGH
self-preempt tid=164028 stint_wake_ratio 16357 HIGH
order-starved comm#8a5f93b2 passover_picks 32 HIGH
held-cpu comm#8a5f93b2 held_ms 36.3994 HIGH
self-preempt tid=164025 stint_wake_ratio 16344 HIGH
hot-waker 164104 wakes_issued 34339 MED
long-range-dependent dispatch-rate hurst 0.879216 MED
migration-avalanche migration-rate runs 38 MED
discord 4438ms profile_vs_mean 1.71107 MED
long-range-dependent migration-rate hurst 0.595689 MED
recvfrom-slow 164073/pand-ipc duration_ms 2.48205 LOW
hot-cpu 3 l2_miss_share 8.9619 LOW
recvfrom-slow 164075/pand-ipc duration_ms 2.48032 LOW
recvfrom-slow 164073/pand-ipc duration_ms 2.32027 LOW
recvfrom-slow 164075/pand-ipc duration_ms 2.31924 LOW
recvfrom-slow 164073/pand-ipc duration_ms 2.24152 LOW
ioctl-slow 164095/python3 duration_ms 0.00106 LOW
ioctl-slow 164095/python3 duration_ms 0.00052 LOW
ioctl-slow 164095/python3 duration_ms 0.00047 LOW
ioctl-slow 164095/python3 duration_ms 0.00044 LOW
ioctl-slow 164095/python3 duration_ms 0.00038 LOW
CAPTURE complete -- 1909707 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 242k wake2run; p50 3us p99 440us p999 1085us worst 2442us; 97.7% fast(<100us) / 2.0% mid / 0.3% tick-floor(>=900us); 0.4% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~203871 distinct values; inversion ratio 0.46
LOCATED: 19 structured region(s) FEW_UNIQUE[3%..4%], FEW_UNIQUE[4%..5%], FEW_UNIQUE[5%..6%], FEW_UNIQUE[6%..6%], FEW_UNIQUE[8%..10%] +14 more
REPORT dispatch-stall
VERDICT: 742 saturated wakes over the 900us floor; PREEMPT-STARVED 41% (0 intervening picks) / ORDER-STARVED 59% (CPU served others first); avg 2.7 pass-overs, p99 10 pass-overs
WORKLOAD ipc-scx_mlfq-16c
THERMAL/POWER
cpu temp peak 55.5 C avg 52.4 C
power avg 117.1 W peak 141.8 W
idle C2 avg 33.5% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
order-starved comm#8a5f93b2 passover_picks 54 HIGH
held-cpu comm#8a5f93b2 held_ms 1.04793 HIGH
self-preempt tid=163708 stint_wake_ratio 28169 HIGH
self-preempt tid=163723 stint_wake_ratio 28039 HIGH
self-preempt tid=163729 stint_wake_ratio 24518 HIGH
self-preempt tid=163727 stint_wake_ratio 22799 HIGH
order-starved comm#8a5f93b2 passover_picks 53 HIGH
held-cpu comm#f7d5fbc2 held_ms 1.00321 HIGH
self-preempt tid=163701 stint_wake_ratio 18509 HIGH
self-preempt tid=163712 stint_wake_ratio 16546 HIGH
self-preempt tid=163711 stint_wake_ratio 15250 HIGH
order-starved comm#328ab0c6 passover_picks 29 HIGH
held-cpu comm#8a5f93b2 held_ms 0.929516 HIGH
self-preempt tid=163719 stint_wake_ratio 15090 HIGH
kthread-strand comm#0dbd5b17 max_strand_ms 8.99985 MED
long-range-dependent dispatch-rate hurst 0.965699 MED
migration-avalanche migration-rate runs 702 MED
discord 3618ms profile_vs_mean 1.7579 MED
hot-cpu 0 l2_miss_share 13.0015 MED
kthread-strand comm#fa4c2f4f max_strand_ms 6.151 MED
kthread-strand comm#da1fb7fd max_strand_ms 6.01023 MED
kthread-strand comm#0878f233 max_strand_ms 5.00493 MED
kthread-strand comm#cfa27126 max_strand_ms 5.00055 MED
long-range-dependent migration-rate hurst 0.7005 MED
recvfrom-slow 163760/pand-ipc duration_ms 2.64224 LOW
recvfrom-slow 163758/pand-ipc duration_ms 2.63363 LOW
recvfrom-slow 163760/pand-ipc duration_ms 2.13918 LOW
recvfrom-slow 163758/pand-ipc duration_ms 2.13913 LOW
recvfrom-slow 163760/pand-ipc duration_ms 2.12039 LOW
ioctl-slow 163781/python3 duration_ms 0.00133 LOW
ioctl-slow 163781/python3 duration_ms 0.00052 LOW
ioctl-slow 163781/python3 duration_ms 0.00052 LOW
ioctl-slow 163781/python3 duration_ms 0.00045 LOW
ioctl-slow 163781/python3 duration_ms 0.00044 LOW
CAPTURE complete -- 1953475 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 262k wake2run; p50 1us p99 127us p999 884us worst 7982us; 98.7% fast(<100us) / 1.2% mid / 0.1% tick-floor(>=900us); 1.6% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~143083 distinct values; inversion ratio 0.41
LOCATED: 59 structured region(s) NEARLY_SORTED[1%..1%], FEW_UNIQUE[1%..2%], FEW_UNIQUE[5%..6%], FEW_UNIQUE[6%..6%], FEW_UNIQUE[7%..7%] +54 more
REPORT dispatch-stall
VERDICT: 245 saturated wakes over the 900us floor; PREEMPT-STARVED 1% (0 intervening picks) / ORDER-STARVED 99% (CPU served others first); avg 10.3 pass-overs, p99 40 pass-overs
REPORT kstrand
VERDICT: 6 strands across 5 per-CPU kthreads; worst HELD strand 9.0ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#0dbd5b17 7 1 9.0 9.0 1 0 comm#c233d180 56%
comm#fa4c2f4f 3 1 6.2 6.2 1 0 comm#8a5f93b2 65%
comm#da1fb7fd 13 1 6.0 6.0 1 0 comm#8a5f93b2 83%
comm#0878f233 0 2 5.0 5.0 2 0 comm#8a5f93b2 100%
comm#cfa27126 15 1 5.0 5.0 1 0 comm#8a5f93b2 100%
WORKLOAD pcpu-burst-16c-EEVDF
THERMAL/POWER
cpu temp peak 59.5 C avg 58.0 C
power avg 141.3 W peak 156.6 W
idle C2 avg 18.6% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
order-starved comm#328ab0c6 passover_picks 10530 HIGH
doublefree-logic 164320/0x0000559e162cbb50 bytes 24 HIGH
self-preempt tid=1047 stint_wake_ratio 19719 HIGH
doublefree-logic 164318/0x000055e8c910fb50 bytes 24 HIGH
doublefree-logic 164319/0x000055b5ba8bdb50 bytes 24 HIGH
self-preempt tid=991 stint_wake_ratio 19693 HIGH
doublefree-logic 164314/0x000055ecee254b50 bytes 24 HIGH
doublefree-logic 164317/0x0000555b65bc3b50 bytes 24 HIGH
self-preempt tid=1635 stint_wake_ratio 19690 HIGH
doublefree-logic 164316/0x000055724820fb50 bytes 24 HIGH
doublefree-logic 164315/0x00005579abf92b50 bytes 24 HIGH
self-preempt tid=867 stint_wake_ratio 19580 HIGH
doublefree-logic 164310/0x00005562d0565b50 bytes 24 HIGH
order-starved comm#328ab0c6 passover_picks 8687 HIGH
doublefree-logic 164308/0x0000559555ec7b50 bytes 24 HIGH
self-preempt tid=1636 stint_wake_ratio 19535 HIGH
doublefree-logic 164307/0x0000558b0ed85b50 bytes 24 HIGH
doublefree-logic 164306/0x000055ed14c49b50 bytes 24 HIGH
self-preempt tid=1634 stint_wake_ratio 19529 HIGH
doublefree-logic 164304/0x00005570691c1b50 bytes 24 HIGH
doublefree-logic 164303/0x000055e73a936b50 bytes 24 HIGH
self-preempt tid=1712 stint_wake_ratio 19521 HIGH
doublefree-logic 164300/0x0000561980453b50 bytes 24 HIGH
doublefree-logic 164299/0x00005588493e7b50 bytes 24 HIGH
order-starved comm#328ab0c6 passover_picks 8251 HIGH
doublefree-logic 164298/0x000055bd05653b50 bytes 24 HIGH
self-preempt tid=1706 stint_wake_ratio 19505 HIGH
long-range-dependent dispatch-rate hurst 0.885633 MED
discord 17304ms profile_vs_mean 1.82068 MED
poll-slow 164309/pandemonium duration_ms 0.00141 LOW
hot-cpu 15 l2_miss_share 7.5324 LOW
poll-slow 164289/pandemonium duration_ms 0.00132 LOW
poll-slow 164311/pandemonium duration_ms 0.00128 LOW
poll-slow 164292/pandemonium duration_ms 0.00124 LOW
poll-slow 164294/pandemonium duration_ms 0.00122 LOW
CAPTURE complete -- 833735 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 81.3k wake2run; p50 1593us p99 13005us p999 19325us worst 29502us; 34.4% fast(<100us) / 6.3% mid / 59.3% tick-floor(>=900us); 0.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~78799 distinct values; inversion ratio 0.38
LOCATED: 2 structured region(s) NEARLY_SORTED[36%..37%], NEARLY_SORTED[84%..84%]
REPORT dispatch-stall
VERDICT: 48.2k saturated wakes over the 900us floor; PREEMPT-STARVED 0% (0 intervening picks) / ORDER-STARVED 100% (CPU served others first); avg 9.6 pass-overs, p99 38 pass-overs
WORKLOAD pcpu-burst-16c-PANDEMONIUM
THERMAL/POWER
cpu temp peak 61.0 C avg 56.6 C
power avg 130.4 W peak 160.0 W
idle C2 avg 33.1% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
order-starved comm#328ab0c6 passover_picks 9878 HIGH
doublefree-logic 169234/0x0000564528d56b50 bytes 24 HIGH
self-preempt tid=1636 stint_wake_ratio 16228 HIGH
doublefree-logic 169232/0x00005563aabd8b50 bytes 24 HIGH
doublefree-logic 169233/0x000056121d4f1b50 bytes 24 HIGH
self-preempt tid=1634 stint_wake_ratio 15664 HIGH
doublefree-logic 169231/0x000055e7325a2b50 bytes 24 HIGH
doublefree-logic 169230/0x0000562984ad8b50 bytes 24 HIGH
self-preempt tid=867 stint_wake_ratio 14881 HIGH
doublefree-logic 169229/0x0000555f458a6b50 bytes 24 HIGH
doublefree-logic 169228/0x000056118d6b4b50 bytes 24 HIGH
self-preempt tid=991 stint_wake_ratio 14818 HIGH
doublefree-logic 169227/0x0000560cab6c4b50 bytes 24 HIGH
order-starved comm#328ab0c6 passover_picks 8911 HIGH
doublefree-logic 169226/0x0000564e2e795b50 bytes 24 HIGH
self-preempt tid=866 stint_wake_ratio 14613 HIGH
doublefree-logic 169225/0x000055fe406d6b50 bytes 24 HIGH
doublefree-logic 169224/0x0000555d04295b50 bytes 24 HIGH
self-preempt tid=1706 stint_wake_ratio 14599 HIGH
doublefree-logic 169222/0x000055694773cb50 bytes 24 HIGH
doublefree-logic 169221/0x0000564ae826cb50 bytes 24 HIGH
self-preempt tid=1712 stint_wake_ratio 14596 HIGH
doublefree-logic 169219/0x0000561f77de1b50 bytes 24 HIGH
doublefree-logic 169217/0x000055c3ee5b6b50 bytes 24 HIGH
order-starved comm#328ab0c6 passover_picks 4649 HIGH
doublefree-logic 169215/0x0000562653eb8b50 bytes 24 HIGH
self-preempt tid=1713 stint_wake_ratio 14559 HIGH
kthread-strand comm#288f4905 max_strand_ms 5.99481 MED
long-range-dependent dispatch-rate hurst 0.949684 MED
discord 5813ms profile_vs_mean 1.54523 MED
kthread-strand comm#288f4086 max_strand_ms 5.99347 MED
kthread-strand comm#12f62b72 max_strand_ms 5.94952 MED
kthread-strand comm#0357d927 max_strand_ms 5.06244 MED
kthread-strand comm#288f4239 max_strand_ms 5.02891 MED
poll-slow 169211/pandemonium duration_ms 0.00177 LOW
hot-cpu 9 l2_miss_share 7.23159 LOW
poll-slow 169203/pandemonium duration_ms 0.00163 LOW
poll-slow 169205/pandemonium duration_ms 0.00149 LOW
poll-slow 169206/pandemonium duration_ms 0.00147 LOW
poll-slow 169204/pandemonium duration_ms 0.00138 LOW
CAPTURE complete -- 856298 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 62.4k wake2run; p50 882us p99 23304us p999 42200us worst 45003us; 39.8% fast(<100us) / 10.7% mid / 49.5% tick-floor(>=900us); 0.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~60428 distinct values; inversion ratio 0.29
LOCATED: 4 structured region(s) NEARLY_SORTED[33%..34%], NEARLY_SORTED[42%..43%], NEARLY_SORTED[53%..53%], NEARLY_SORTED[71%..71%]
REPORT dispatch-stall
VERDICT: 30.9k saturated wakes over the 900us floor; PREEMPT-STARVED 0% (0 intervening picks) / ORDER-STARVED 100% (CPU served others first); avg 14.3 pass-overs, p99 78 pass-overs
REPORT kstrand
VERDICT: 6 strands across 5 per-CPU kthreads; worst HELD strand 6.0ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#288f4905 4 1 6.0 6.0 1 0 comm#7e44bede 17%
comm#288f4086 1 1 6.0 6.0 1 0 comm#8a5f93b2 33%
comm#12f62b72 6 1 5.9 5.9 1 0 comm#8a5f93b2 17%
comm#0357d927 1 2 5.1 5.1 2 0 comm#8a5f93b2 39%
comm#288f4239 0 1 5.0 5.0 1 0 comm#8a5f93b2 20%
WORKLOAD pcpu-burst-16c-scx_cake
THERMAL/POWER
cpu temp peak 59.5 C avg 57.6 C
power avg 140.3 W peak 152.4 W
idle C2 avg 15.2% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
order-starved comm#c233d180 passover_picks 1491 HIGH
doublefree-logic 168111/0x000055db9a97db50 bytes 24 HIGH
self-preempt tid=866 stint_wake_ratio 22200 HIGH
doublefree-logic 168110/0x000055f429511b50 bytes 24 HIGH
doublefree-logic 168109/0x00005648651f1b50 bytes 24 HIGH
self-preempt tid=867 stint_wake_ratio 22059 HIGH
doublefree-logic 168108/0x0000562f3de8db50 bytes 24 HIGH
doublefree-logic 168107/0x00005639da140b50 bytes 24 HIGH
self-preempt tid=1047 stint_wake_ratio 17418 HIGH
doublefree-logic 168106/0x0000559cd3657b50 bytes 24 HIGH
doublefree-logic 168105/0x000055d801e14b50 bytes 24 HIGH
self-preempt tid=1706 stint_wake_ratio 17354 HIGH
doublefree-logic 168104/0x000056178b14bb50 bytes 24 HIGH
order-starved comm#5a4f434f passover_picks 1226 HIGH
doublefree-logic 168103/0x000055d8d8713b50 bytes 24 HIGH
self-preempt tid=1635 stint_wake_ratio 16977 HIGH
doublefree-logic 168102/0x0000556f17a88b50 bytes 24 HIGH
doublefree-logic 168101/0x0000561b89161b50 bytes 24 HIGH
self-preempt tid=1637 stint_wake_ratio 16941 HIGH
doublefree-logic 168100/0x0000556426025b50 bytes 24 HIGH
doublefree-logic 168099/0x000055c16e89cb50 bytes 24 HIGH
self-preempt tid=1707 stint_wake_ratio 12980 HIGH
doublefree-logic 168098/0x000055f5fa212b50 bytes 24 HIGH
doublefree-logic 168097/0x0000560a99e1ab50 bytes 24 HIGH
order-starved comm#c233d180 passover_picks 973 HIGH
doublefree-logic 168095/0x00005627fcb95b50 bytes 24 HIGH
self-preempt tid=1711 stint_wake_ratio 12942 HIGH
kthread-strand comm#0878f080 max_strand_ms 40.9992 MED
long-range-dependent dispatch-rate hurst 0.850393 MED
hot-cpu 3 l2_miss_share 14.8383 MED
kthread-strand comm#0dbd5b17 max_strand_ms 35.9979 MED
kthread-strand comm#1beaeefc max_strand_ms 32.9996 MED
kthread-strand comm#0357d927 max_strand_ms 32.9945 MED
kthread-strand comm#2f09aaa5 max_strand_ms 31.9986 MED
kthread-strand comm#9f52aa6a max_strand_ms 22.0004 MED
kthread-strand comm#180cd039 max_strand_ms 21.9997 MED
kthread-strand comm#d4e6d6a4 max_strand_ms 21.9985 MED
kthread-strand comm#12f62b72 max_strand_ms 21.9963 MED
kthread-strand comm#ca814de8 max_strand_ms 20.9996 MED
kthread-strand comm#dca24006 max_strand_ms 20.9993 MED
kthread-strand comm#e3146def max_strand_ms 20 MED
kthread-strand comm#288f459f max_strand_ms 19.9674 MED
kthread-strand comm#da1fb7fd max_strand_ms 18.9996 MED
kthread-strand comm#fa4c2f4f max_strand_ms 10.9997 MED
kthread-strand comm#df364860 max_strand_ms 10.9955 MED
kthread-strand comm#288f3d20 max_strand_ms 10.0157 MED
kthread-strand comm#2f099b5a max_strand_ms 9.0009 MED
kthread-strand comm#288f4239 max_strand_ms 5.05662 MED
poll-slow 168087/pandemonium duration_ms 0.00181 LOW
poll-slow 168081/pandemonium duration_ms 0.00134 LOW
poll-slow 168093/pandemonium duration_ms 0.00131 LOW
poll-slow 168091/pandemonium duration_ms 0.00127 LOW
poll-slow 168084/pandemonium duration_ms 0.00126 LOW
CAPTURE complete -- 609725 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 99.9k wake2run; p50 15us p99 26999us p999 42011us worst 75001us; 56.5% fast(<100us) / 2.0% mid / 41.5% tick-floor(>=900us); 0.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~98371 distinct values; inversion ratio 0.41
REPORT dispatch-stall
VERDICT: 41.5k saturated wakes over the 900us floor; PREEMPT-STARVED 0% (0 intervening picks) / ORDER-STARVED 100% (CPU served others first); avg 3.0 pass-overs, p99 15 pass-overs
REPORT kstrand
VERDICT: 159 strands across 19 per-CPU kthreads; worst HELD strand 41.0ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#0878f080 0 5 41.0 41.0 5 0 comm#8a5f93b2 51%
comm#0dbd5b17 7 16 36.0 36.0 16 0 comm#8a5f93b2 58%
comm#1beaeefc 4 16 33.0 33.0 16 0 comm#8a5f93b2 64%
comm#0357d927 1 23 33.0 33.0 23 0 comm#8a5f93b2 64%
comm#2f09aaa5 2 20 32.0 32.0 20 0 comm#8a5f93b2 66%
comm#9f52aa6a 8 8 22.0 22.0 8 0 comm#8a5f93b2 95%
comm#180cd039 5 20 22.0 22.0 20 0 comm#8a5f93b2 95%
comm#d4e6d6a4 12 4 22.0 22.0 4 0 comm#8a5f93b2 95%
comm#12f62b72 6 12 22.0 22.0 12 0 comm#8a5f93b2 95%
comm#ca814de8 14 5 21.0 21.0 5 0 comm#8a5f93b2 100%
comm#dca24006 9 5 21.0 21.0 5 0 comm#8a5f93b2 100%
comm#e3146def 11 5 20.0 20.0 5 0 comm#8a5f93b2 100%
comm#288f459f 6 4 20.0 20.0 4 0 comm#8a5f93b2 100%
comm#da1fb7fd 13 4 19.0 19.0 4 0 comm#8a5f93b2 100%
comm#fa4c2f4f 3 6 11.0 11.0 6 0 comm#8a5f93b2 24%
comm#df364860 10 3 11.0 11.0 3 0 comm#8a5f93b2 100%
comm#288f3d20 3 1 10.0 10.0 1 0 comm#8a5f93b2 75%
comm#2f099b5a 2 1 9.0 9.0 1 0 comm#8a5f93b2 100%
comm#288f4239 0 1 5.1 5.1 1 0 comm#8a5f93b2 40%
WORKLOAD pcpu-burst-16c-scx_mlfq
THERMAL/POWER
cpu temp peak 60.1 C avg 56.8 C
power avg 133.1 W peak 155.9 W
idle C2 avg 26.8% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
order-starved comm#328ab0c6 passover_picks 3067 HIGH
kthread-strand comm#cfa26f73 max_strand_ms 564.999 HIGH
doublefree-logic 166043/0x00005588411ea680 bytes 240 HIGH
self-preempt tid=1712 stint_wake_ratio 16592 HIGH
kthread-strand comm#288f4e1e max_strand_ms 113.005 HIGH
kthread-strand comm#cfa27126 max_strand_ms 104.003 HIGH
doublefree-logic 166058/0x0000556baeb9ab50 bytes 24 HIGH
doublefree-logic 166057/0x00005588411ecb50 bytes 24 HIGH
self-preempt tid=991 stint_wake_ratio 16586 HIGH
doublefree-logic 166056/0x0000562960e45b50 bytes 24 HIGH
doublefree-logic 166055/0x00005652726f0b50 bytes 24 HIGH
self-preempt tid=1635 stint_wake_ratio 16537 HIGH
doublefree-logic 166054/0x0000561a9c16ab50 bytes 24 HIGH
doublefree-logic 166053/0x0000562dd14bab50 bytes 24 HIGH
self-preempt tid=1634 stint_wake_ratio 16502 HIGH
doublefree-logic 166051/0x0000559deba32b50 bytes 24 HIGH
order-starved comm#5a4f434f passover_picks 2763 HIGH
doublefree-logic 166050/0x0000564cc61c8b50 bytes 24 HIGH
doublefree-logic 166049/0x0000556ec3965b50 bytes 24 HIGH
self-preempt tid=1713 stint_wake_ratio 16501 HIGH
doublefree-logic 166048/0x00005652e3de6b50 bytes 24 HIGH
doublefree-logic 166047/0x0000563c98a3db50 bytes 24 HIGH
self-preempt tid=1707 stint_wake_ratio 16414 HIGH
doublefree-logic 166045/0x00005644c57fab50 bytes 24 HIGH
doublefree-logic 166042/0x000055f7e525ab50 bytes 24 HIGH
self-preempt tid=867 stint_wake_ratio 16398 HIGH
doublefree-logic 166040/0x0000560ca68b8b50 bytes 24 HIGH
doublefree-logic 166038/0x000055ceb2f56b50 bytes 24 HIGH
order-starved comm#328ab0c6 passover_picks 2762 HIGH
doublefree-logic 166037/0x0000555c1bf61b50 bytes 24 HIGH
self-preempt tid=1047 stint_wake_ratio 16384 HIGH
long-range-dependent dispatch-rate hurst 0.883763 MED
discord 7428ms profile_vs_mean 1.69662 MED
kthread-strand comm#12f62b72 max_strand_ms 85.1934 MED
kthread-strand comm#d1030141 max_strand_ms 83.0256 MED
kthread-strand comm#12f62d25 max_strand_ms 79.0258 MED
kthread-strand comm#e3146def max_strand_ms 79.0152 MED
kthread-strand comm#df364860 max_strand_ms 78.0265 MED
kthread-strand comm#0357d927 max_strand_ms 77.0555 MED
kthread-strand comm#eb6af029 max_strand_ms 76.0959 MED
kthread-strand comm#ca814de8 max_strand_ms 75.9766 MED
kthread-strand comm#0878f080 max_strand_ms 74.016 MED
kthread-strand comm#1beaeefc max_strand_ms 73.0337 MED
kthread-strand comm#d4e6d6a4 max_strand_ms 69.9285 MED
kthread-strand comm#0dbd5b17 max_strand_ms 68.5049 MED
kthread-strand comm#9f52aa6a max_strand_ms 67.9945 MED
kthread-strand comm#dca24006 max_strand_ms 67.3492 MED
kthread-strand comm#2f09aaa5 max_strand_ms 63.016 MED
kthread-strand comm#180cd039 max_strand_ms 63.0105 MED
kthread-strand comm#dbbca7d3 max_strand_ms 54.0153 MED
kthread-strand comm#eb6aee76 max_strand_ms 51.9887 MED
kthread-strand comm#da1fb7fd max_strand_ms 51.0253 MED
kthread-strand comm#288f4086 max_strand_ms 43.0938 MED
kthread-strand comm#288f4239 max_strand_ms 40.0049 MED
kthread-strand comm#19c571dd max_strand_ms 38.0104 MED
kthread-strand comm#288f4752 max_strand_ms 37.9902 MED
kthread-strand comm#eb6af6f5 max_strand_ms 36.9876 MED
kthread-strand comm#288f43ec max_strand_ms 35.9796 MED
kthread-strand comm#2f099b5a max_strand_ms 35.0291 MED
kthread-strand comm#a3cc20ce max_strand_ms 35.0041 MED
kthread-strand comm#eb6af38f max_strand_ms 33.0011 MED
kthread-strand comm#288f3ed3 max_strand_ms 23.9961 MED
kthread-strand comm#fa4c2f4f max_strand_ms 23.2282 MED
kthread-strand comm#288f4fd1 max_strand_ms 21.4324 MED
kthread-strand comm#ddbfab61 max_strand_ms 21.0024 MED
kthread-strand comm#c4395182 max_strand_ms 19.5104 MED
kthread-strand comm#4f7aa3e7 max_strand_ms 18.2099 MED
kthread-strand comm#eb6af542 max_strand_ms 17.0043 MED
kthread-strand comm#eb6af1dc max_strand_ms 14.0216 MED
kthread-strand comm#288f3d20 max_strand_ms 11.529 MED
kthread-strand comm#658123d8 max_strand_ms 11.0123 MED
kthread-strand comm#288f459f max_strand_ms 6.84078 MED
mean-reverting migration-rate hurst -2.29195 MED
poll-slow 166034/pandemonium duration_ms 0.00192 LOW
hot-cpu 3 l2_miss_share 7.89312 LOW
poll-slow 166027/pandemonium duration_ms 0.00176 LOW
poll-slow 166028/pandemonium duration_ms 0.00135 LOW
poll-slow 166039/pandemonium duration_ms 0.00129 LOW
poll-slow 166033/pandemonium duration_ms 0.00129 LOW
CAPTURE complete -- 495317 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 25.6k wake2run; p50 19us p99 61351us p999 118377us worst 1000284us; 56.5% fast(<100us) / 2.0% mid / 41.5% tick-floor(>=900us); 0.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~25171 distinct values; inversion ratio 0.45
LOCATED: 1 structured region(s) NEARLY_SORTED[80%..82%]
REPORT dispatch-stall
VERDICT: 10.6k saturated wakes over the 900us floor; PREEMPT-STARVED 0% (0 intervening picks) / ORDER-STARVED 100% (CPU served others first); avg 22.7 pass-overs, p99 125 pass-overs
REPORT kstrand
VERDICT: 885 strands across 42 per-CPU kthreads; worst HELD strand 150.0ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#cfa26f73 15 66 565.0 565.0 65 1 comm#c233d180 75%
comm#288f4e1e 9 5 113.0 113.0 5 0 comm#8a5f93b2 4%
comm#cfa27126 15 37 104.0 104.0 37 0 comm#8a5f93b2 10%
comm#12f62b72 6 37 85.2 85.2 37 0 comm#8a5f93b2 6%
comm#d1030141 15 2 83.0 83.0 2 0 comm#8a5f93b2 12%
comm#12f62d25 6 7 79.0 79.0 7 0 comm#8a5f93b2 6%
comm#e3146def 11 38 79.0 79.0 38 0 comm#8a5f93b2 6%
comm#df364860 10 56 78.0 78.0 56 0 comm#8a5f93b2 6%
comm#0357d927 1 51 77.1 77.1 51 0 comm#8a5f93b2 6%
comm#eb6af029 15 12 76.1 76.1 12 0 comm#8a5f93b2 13%
comm#ca814de8 14 44 76.0 76.0 44 0 comm#8a5f93b2 7%
comm#0878f080 0 78 74.0 74.0 78 0 comm#8a5f93b2 7%
comm#1beaeefc 4 56 73.0 73.0 56 0 comm#8a5f93b2 7%
comm#d4e6d6a4 12 46 69.9 69.9 46 0 comm#8a5f93b2 7%
comm#0dbd5b17 7 41 68.5 68.5 41 0 comm#8a5f93b2 14%
comm#9f52aa6a 8 39 68.0 68.0 39 0 comm#8a5f93b2 7%
comm#dca24006 9 25 67.3 67.3 25 0 comm#8a5f93b2 7%
comm#2f09aaa5 2 61 63.0 63.0 61 0 comm#8a5f93b2 8%
comm#180cd039 5 52 63.0 63.0 52 0 comm#8a5f93b2 8%
comm#dbbca7d3 11 7 54.0 54.0 7 0 comm#8a5f93b2 9%
WORKLOAD sojourn-pressure-16c-EEVDF
THERMAL/POWER
cpu temp peak 59.2 C avg 55.7 C
power avg 132.1 W peak 155.2 W
idle C2 avg 36.2% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
order-starved comm#328ab0c6 passover_picks 332 HIGH
held-cpu comm#8a5f93b2 held_ms 0.948775 HIGH
self-preempt tid=1636 stint_wake_ratio 13476 HIGH
self-preempt tid=1712 stint_wake_ratio 13466 HIGH
self-preempt tid=1634 stint_wake_ratio 13466 HIGH
self-preempt tid=1637 stint_wake_ratio 13463 HIGH
order-starved comm#328ab0c6 passover_picks 65 HIGH
self-preempt tid=1047 stint_wake_ratio 13455 HIGH
self-preempt tid=1713 stint_wake_ratio 13453 HIGH
self-preempt tid=1710 stint_wake_ratio 13446 HIGH
order-starved comm#328ab0c6 passover_picks 59 HIGH
self-preempt tid=991 stint_wake_ratio 13446 HIGH
hot-cpu 6 l2_miss_share 10.3391 MED
CAPTURE complete -- 541436 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 3.8k wake2run; p50 4us p99 2486us p999 3329us worst 3989us; 79.2% fast(<100us) / 9.5% mid / 11.3% tick-floor(>=900us); 0.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~3403 distinct values; inversion ratio 0.53
REPORT dispatch-stall
VERDICT: 425 saturated wakes over the 900us floor; PREEMPT-STARVED 0% (0 intervening picks) / ORDER-STARVED 100% (CPU served others first); avg 5.5 pass-overs, p99 13 pass-overs
WORKLOAD sojourn-pressure-16c-PANDEMONIUM
THERMAL/POWER
cpu temp peak 59.0 C avg 57.0 C
power avg 124.9 W peak 156.5 W
idle C2 avg 51.6% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
order-starved comm#8a5f93b2 passover_picks 38 HIGH
held-cpu comm#8a5f93b2 held_ms 0.937763 HIGH
self-preempt tid=1635 stint_wake_ratio 11193 HIGH
self-preempt tid=1047 stint_wake_ratio 10980 HIGH
self-preempt tid=1711 stint_wake_ratio 10229 HIGH
self-preempt tid=867 stint_wake_ratio 10225 HIGH
order-starved comm#8a5f93b2 passover_picks 38 HIGH
held-cpu comm#8a5f93b2 held_ms 0.936473 HIGH
self-preempt tid=1706 stint_wake_ratio 9843 HIGH
self-preempt tid=1712 stint_wake_ratio 9832 HIGH
self-preempt tid=1636 stint_wake_ratio 9650 HIGH
order-starved comm#8a5f93b2 passover_picks 36 HIGH
held-cpu comm#8a5f93b2 held_ms 0.934504 HIGH
self-preempt tid=866 stint_wake_ratio 9601 HIGH
kthread-strand comm#12f62b72 max_strand_ms 5.00345 MED
long-range-dependent migration-rate hurst 0.542983 MED
kthread-strand comm#2f09aaa5 max_strand_ms 5.00086 MED
hot-cpu 2 l2_miss_share 9.15254 LOW
CAPTURE complete -- 537874 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 2.9k wake2run; p50 583us p99 2934us p999 3933us worst 4560us; 37.6% fast(<100us) / 30.6% mid / 31.8% tick-floor(>=900us); 0.1% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~2806 distinct values; inversion ratio 0.59
REPORT dispatch-stall
VERDICT: 906 saturated wakes over the 900us floor; PREEMPT-STARVED 1% (0 intervening picks) / ORDER-STARVED 99% (CPU served others first); avg 3.0 pass-overs, p99 5 pass-overs
REPORT kstrand
VERDICT: 2 strands across 2 per-CPU kthreads; worst HELD strand 5.0ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#12f62b72 6 1 5.0 5.0 1 0 comm#8a5f93b2 40%
comm#2f09aaa5 2 1 5.0 5.0 1 0 comm#8a5f93b2 35%
WORKLOAD sojourn-pressure-16c-scx_cake
THERMAL/POWER
cpu temp peak 59.5 C avg 52.0 C
power avg 107.3 W peak 157.0 W
idle C2 avg 59.4% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
order-starved comm#d9d17ccd passover_picks 84 HIGH
held-cpu comm#8a5f93b2 held_ms 9.08213 HIGH
self-preempt tid=1106 stint_wake_ratio 8966 HIGH
hot-cpu 15 l2_miss_share 27.3716 HIGH
self-preempt tid=866 stint_wake_ratio 8602 HIGH
self-preempt tid=1047 stint_wake_ratio 8457 HIGH
self-preempt tid=867 stint_wake_ratio 8456 HIGH
order-starved comm#328ab0c6 passover_picks 72 HIGH
held-cpu comm#8a5f93b2 held_ms 1.94025 HIGH
self-preempt tid=1096 stint_wake_ratio 8452 HIGH
self-preempt tid=1707 stint_wake_ratio 8433 HIGH
self-preempt tid=1706 stint_wake_ratio 8429 HIGH
order-starved comm#328ab0c6 passover_picks 56 HIGH
held-cpu comm#8a5f93b2 held_ms 1.02872 HIGH
self-preempt tid=991 stint_wake_ratio 8427 HIGH
kthread-strand comm#0878f080 max_strand_ms 38.9991 MED
long-range-dependent dispatch-rate hurst 0.613938 MED
kthread-strand comm#9f52aa6a max_strand_ms 31.9941 MED
kthread-strand comm#fa4c2f4f max_strand_ms 23.0009 MED
kthread-strand comm#da1fb7fd max_strand_ms 21.9991 MED
kthread-strand comm#180cd039 max_strand_ms 21.9991 MED
kthread-strand comm#e3146def max_strand_ms 21.999 MED
kthread-strand comm#2f09aaa5 max_strand_ms 20.9997 MED
kthread-strand comm#1beaeefc max_strand_ms 20.9992 MED
kthread-strand comm#df364860 max_strand_ms 20.9991 MED
kthread-strand comm#ca814de8 max_strand_ms 20.9897 MED
kthread-strand comm#0357d927 max_strand_ms 17.9998 MED
kthread-strand comm#12f62b72 max_strand_ms 17.9993 MED
kthread-strand comm#0dbd5b17 max_strand_ms 17.9989 MED
kthread-strand comm#dca24006 max_strand_ms 14.9994 MED
kthread-strand comm#d4e6d6a4 max_strand_ms 12.9996 MED
kthread-strand comm#cfa26f73 max_strand_ms 12.9993 MED
kthread-strand comm#eb6af029 max_strand_ms 6.99271 MED
CAPTURE complete -- 381768 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 2.5k wake2run; p50 35us p99 12296us p999 15936us worst 21178us; 68.3% fast(<100us) / 8.4% mid / 23.3% tick-floor(>=900us); 0.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~2441 distinct values; inversion ratio 0.60
REPORT dispatch-stall
VERDICT: 577 saturated wakes over the 900us floor; PREEMPT-STARVED 1% (0 intervening picks) / ORDER-STARVED 99% (CPU served others first); avg 16.4 pass-overs, p99 28 pass-overs
REPORT kstrand
VERDICT: 72 strands across 17 per-CPU kthreads; worst HELD strand 39.0ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#0878f080 0 13 39.0 39.0 13 0 comm#8a5f93b2 54%
comm#9f52aa6a 8 3 32.0 32.0 3 0 comm#8a5f93b2 66%
comm#fa4c2f4f 3 5 23.0 23.0 5 0 comm#8a5f93b2 100%
comm#da1fb7fd 13 3 22.0 22.0 3 0 comm#8a5f93b2 95%
comm#180cd039 5 4 22.0 22.0 4 0 comm#8a5f93b2 95%
comm#e3146def 11 3 22.0 22.0 3 0 comm#8a5f93b2 95%
comm#2f09aaa5 2 3 21.0 21.0 3 0 comm#8a5f93b2 100%
comm#1beaeefc 4 6 21.0 21.0 6 0 comm#8a5f93b2 100%
comm#df364860 10 2 21.0 21.0 2 0 comm#8a5f93b2 100%
comm#ca814de8 14 3 21.0 21.0 3 0 comm#8a5f93b2 100%
comm#0357d927 1 9 18.0 18.0 9 0 comm#8a5f93b2 100%
comm#12f62b72 6 2 18.0 18.0 2 0 comm#8a5f93b2 100%
comm#0dbd5b17 7 5 18.0 18.0 5 0 comm#8a5f93b2 100%
comm#dca24006 9 4 15.0 15.0 4 0 comm#8a5f93b2 100%
comm#d4e6d6a4 12 4 13.0 13.0 4 0 comm#8a5f93b2 100%
comm#cfa26f73 15 1 13.0 13.0 1 0 comm#8a5f93b2 100%
comm#eb6af029 15 2 7.0 7.0 2 0 comm#c233d180 59%
WORKLOAD sojourn-pressure-16c-scx_mlfq
THERMAL/POWER
cpu temp peak 59.0 C avg 53.4 C
power avg 115.7 W peak 154.2 W
idle C2 avg 56.2% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
order-starved comm#328ab0c6 passover_picks 2 HIGH
kthread-strand comm#cfa27126 max_strand_ms 232.009 HIGH
self-preempt tid=1047 stint_wake_ratio 12983 HIGH
self-preempt tid=991 stint_wake_ratio 12981 HIGH
self-preempt tid=1707 stint_wake_ratio 12975 HIGH
self-preempt tid=866 stint_wake_ratio 12973 HIGH
order-starved comm#328ab0c6 passover_picks 2 HIGH
self-preempt tid=1711 stint_wake_ratio 12963 HIGH
self-preempt tid=1636 stint_wake_ratio 12960 HIGH
self-preempt tid=1706 stint_wake_ratio 12951 HIGH
order-starved comm#8a5f93b2 passover_picks 1 HIGH
self-preempt tid=867 stint_wake_ratio 12945 HIGH
long-range-dependent migration-rate hurst 0.577529 MED
discord 7724ms profile_vs_mean 1.55312 MED
kthread-strand comm#e3146def max_strand_ms 74.0121 MED
kthread-strand comm#0878f233 max_strand_ms 61.0091 MED
kthread-strand comm#0357d774 max_strand_ms 59.7254 MED
kthread-strand comm#2f09aaa5 max_strand_ms 54.0044 MED
kthread-strand comm#0dbd5b17 max_strand_ms 50.004 MED
kthread-strand comm#da1faa65 max_strand_ms 49.9873 MED
kthread-strand comm#1beaeefc max_strand_ms 44.0384 MED
kthread-strand comm#0357d927 max_strand_ms 40.012 MED
kthread-strand comm#d4e6d6a4 max_strand_ms 40.003 MED
kthread-strand comm#0878f080 max_strand_ms 39.9974 MED
kthread-strand comm#180cd039 max_strand_ms 38.0051 MED
kthread-strand comm#eb6af38f max_strand_ms 35.0024 MED
kthread-strand comm#288f3ed3 max_strand_ms 30.0087 MED
kthread-strand comm#da1fb7fd max_strand_ms 30.0064 MED
kthread-strand comm#d4e6d4f1 max_strand_ms 30.0045 MED
kthread-strand comm#fa4c2f4f max_strand_ms 21.2512 MED
kthread-strand comm#df364860 max_strand_ms 21.0086 MED
kthread-strand comm#ca814de8 max_strand_ms 21.0057 MED
kthread-strand comm#d4e6d33e max_strand_ms 20.0063 MED
kthread-strand comm#da1fa8b2 max_strand_ms 19.9981 MED
kthread-strand comm#9f52aa6a max_strand_ms 19.0119 MED
kthread-strand comm#eb6af029 max_strand_ms 17.9917 MED
kthread-strand comm#288f4239 max_strand_ms 8.99126 MED
kthread-strand comm#12f62b72 max_strand_ms 7.00831 MED
kthread-strand comm#180ccb20 max_strand_ms 5.00436 MED
mean-reverting dispatch-rate hurst 0.333285 MED
hot-cpu 9 l2_miss_share 8.06245 LOW
CAPTURE complete -- 404516 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 4.4k wake2run; p50 2us p99 162us p999 1360us worst 3815us; 95.8% fast(<100us) / 4.1% mid / 0.1% tick-floor(>=900us); 1.7% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~3497 distinct values; inversion ratio 0.55
REPORT dispatch-stall
VERDICT: 6 saturated wakes over the 900us floor; PREEMPT-STARVED 0% (0 intervening picks) / ORDER-STARVED 100% (CPU served others first); avg 8.3 pass-overs, p99 30 pass-overs
REPORT kstrand
VERDICT: 162 strands across 26 per-CPU kthreads; worst HELD strand 232.0ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#cfa27126 15 18 232.0 232.0 18 0 comm#8a5f93b2 4%
comm#e3146def 11 4 74.0 74.0 4 0 comm#8a5f93b2 7%
comm#0878f233 0 25 61.0 61.0 25 0 comm#8a5f93b2 8%
comm#0357d774 1 4 59.7 59.7 4 0 comm#8a5f93b2 8%
comm#2f09aaa5 2 7 54.0 54.0 7 0 comm#8a5f93b2 9%
comm#0dbd5b17 7 4 50.0 50.0 4 0 comm#8a5f93b2 10%
comm#da1faa65 13 3 50.0 50.0 3 0 comm#8a5f93b2 10%
comm#1beaeefc 4 2 44.0 44.0 2 0 comm#8a5f93b2 11%
comm#0357d927 1 15 40.0 40.0 15 0 comm#8a5f93b2 13%
comm#d4e6d6a4 12 14 40.0 40.0 14 0 comm#8a5f93b2 13%
comm#0878f080 0 9 40.0 40.0 9 0 comm#8a5f93b2 13%
comm#180cd039 5 8 38.0 38.0 8 0 comm#8a5f93b2 13%
comm#eb6af38f 13 1 35.0 35.0 1 0 comm#8a5f93b2 14%
comm#288f3ed3 2 1 30.0 30.0 1 0 comm#8a5f93b2 17%
comm#da1fb7fd 13 10 30.0 30.0 10 0 comm#8a5f93b2 17%
comm#d4e6d4f1 12 3 30.0 30.0 3 0 comm#8a5f93b2 17%
comm#fa4c2f4f 3 12 21.3 21.3 12 0 comm#8a5f93b2 24%
comm#df364860 10 3 21.0 21.0 3 0 comm#8a5f93b2 24%
comm#ca814de8 14 4 21.0 21.0 4 0 comm#8a5f93b2 24%
comm#d4e6d33e 12 2 20.0 20.0 2 0 comm#8a5f93b2 25%
