https://pastein.ru/t/-t0
Загрузка данных
PRISM v5.21.0 [9661e39] [sched=all-scx]
captured: 20260910-212032
SCHEDULER STABILITY
EJECTED pandemonium -- "[20034.781024] sched_ext: BPF scheduler 'scx_flow_3.1.2_x86_64_unknown_linux_gnu' disabled (runnable task stall)"
CLEAN-ROOM
state NOISY -- load 0.6/16cpu, uptime 5.0h
SYSTEM
cpu AMD Ryzen 9 5950X 16-Core Processor (16c/16t)
memory 31,2 GiB
gpu amdgpu (1002:747E)
kernel 7.2.3-1-cachyos
scheduler EEVDF
STALL SUSCEPTIBILITY
kernel 7.2.3-1-cachyos
tick HZ=1000 NO_HZ_FULL preempt=PREEMPT_DYNAMIC
cpus 16 governor=performance deepest-idle=C2 (3 states)
cmdline nohz_full=- isolcpus=-
verdict score 4 -> MODERATE
factors HZ=1000 (tight ticks -- narrows window); idle CPUs go tickless (NO_HZ_IDLE behavior; NO_HZ_FULL built but no nohz_full= mask, so unused); 16 CPUs (many idle at once)
COMPARISON (p99 wake2run, lower is better)
fork-thread EEVDF 40860us -> BPF 4897us (8.3x better), ADAPTIVE 5030us (8.1x better)
ipc EEVDF 25us -> BPF 12us (2.1x better), ADAPTIVE 12us (2.1x better)
pcpu-burst EEVDF 14504us -> PANDEMONIUM 12391us (1.2x better)
sojourn-pressure EEVDF 4284us -> PANDEMONIUM 872us (4.9x better)
cachyos EEVDF 1633us -> BPF 721us (2.3x better), ADAPTIVE 701us (2.3x better)
POPULATION (montauk cross-run statistics over 6 run(s), per workload)
cachyos-primes eevdf vs pandemonium-adaptive: N=2 cliff +1.00 perm p=0.036 power n=3
cachyos-primes eevdf vs pandemonium-bpf: N=2 cliff +1.00 perm p=0.036 power n=3
cachyos-primes pandemonium-adaptive vs pandemonium-bpf: N=6 cliff +0.06 perm p=0.916 power n>20
cachyos-stress-ng-cpu-cache-mem eevdf vs pandemonium-adaptive: N=2 cliff +1.00 perm p=0.036 power n=4
cachyos-stress-ng-cpu-cache-mem eevdf vs pandemonium-bpf: N=2 cliff +1.00 perm p=0.036 power n=3
cachyos-stress-ng-cpu-cache-mem pandemonium-adaptive vs pandemonium-bpf: N=6 cliff +0.50 perm p=0.258 power n>20
cachyos-xz-compression eevdf vs pandemonium-adaptive: N=2 cliff +1.00 perm p=0.036 power n=3
cachyos-xz-compression eevdf vs pandemonium-bpf: N=2 cliff +1.00 perm p=0.036 power n=3
cachyos-xz-compression pandemonium-adaptive vs pandemonium-bpf: N=6 cliff +0.11 perm p=0.701 power n>20
cold-wake-16c eevdf vs pandemonium: N=2 cliff -0.75 perm p=0.267 power n=6
cold-wake-starve-16c eevdf vs pandemonium: N=2 cliff +1.00 perm p=0.067 power n=4
fork-thread eevdf vs pandemonium-adaptive: N=2 cliff +1.00 perm p=0.036 power n=2
fork-thread eevdf vs pandemonium-bpf: N=2 cliff +1.00 perm p=0.036 power n=3
fork-thread pandemonium-adaptive vs pandemonium-bpf: N=6 cliff +0.06 perm p=0.881 power n>20
ipc-16c eevdf vs pandemonium-adaptive: N=2 cliff +0.00 perm p=0.733 power n=20
ipc-16c eevdf vs pandemonium-bpf: N=2 cliff +0.00 perm p=0.867 power n>20
ipc-16c pandemonium-adaptive vs pandemonium-bpf: N=4 cliff +0.06 perm p=0.914 power n>20
ipc-32c pandemonium-adaptive vs pandemonium-bpf: N=2 cliff +1.00 perm p=0.333 power n=3
pcpu-burst-16c eevdf vs pandemonium: N=2 cliff -0.25 perm p=0.400 power n=14
sojourn-pressure-16c eevdf vs pandemonium: N=2 cliff +0.75 perm p=0.200 power n=6
cachyos-primes MCB best (lower p99): pandemonium-adaptive (tied), pandemonium-bpf (tied)
cachyos-stress-ng-cpu-cache-mem MCB best (lower p99): pandemonium-adaptive (tied), pandemonium-bpf (tied)
cachyos-xz-compression MCB best (lower p99): pandemonium-adaptive (tied), pandemonium-bpf (tied)
cold-wake-16c MCB best (lower p99): eevdf
cold-wake-starve-16c MCB best (lower p99): pandemonium
fork-thread MCB best (lower p99): pandemonium-adaptive (tied), pandemonium-bpf (tied)
ipc-16c MCB best (lower p99): eevdf (tied), pandemonium-adaptive (tied), pandemonium-bpf (tied)
ipc-32c MCB best (lower p99): pandemonium-bpf
pcpu-burst-16c MCB best (lower p99): eevdf (tied), pandemonium (tied)
sojourn-pressure-16c MCB best (lower p99): pandemonium
N<3 per arm -- montauk reports the effect size but the inference is underpowered; repeat runs accumulate here
STORM (cpu_release flood, powersave -- lower storm% is better)
REAL IPI storm (preempt-kick dominant); storm 68/73 intervals (93.2%); reenq/s p50=130893 peak=135104; kick/s=244085 (preempt 122957) reenq/s=122974
WORKLOAD cachyos-EEVDF-primes
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
hot-cpu 15 l2_miss_share 33.4716 HIGH
long-range-dependent dispatch-rate hurst 0.620668 MED
long-range-dependent migration-rate hurst 0.588268 MED
CAPTURE complete -- 11923 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 7.0k wake2run; p50 5us p99 1702us p999 3233us worst 3328us; 93.6% fast(<100us) / 1.3% mid / 5.1% tick-floor(>=900us); 0.2% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~5648 distinct values; inversion ratio 0.48
WORKLOAD cachyos-EEVDF-stress-ng-cpu-cache-mem
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
long-range-dependent dispatch-rate hurst 0.819376 MED
migration-avalanche migration-rate runs 5 MED
discord 7128ms profile_vs_mean 1.92344 MED
long-range-dependent migration-rate hurst 0.630848 MED
hot-cpu 15 l2_miss_share 7.06979 LOW
CAPTURE complete -- 105193 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 98.6k wake2run; p50 3us p99 1617us p999 3234us worst 6994us; 96.4% fast(<100us) / 1.2% mid / 2.4% tick-floor(>=900us); 0.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~70881 distinct values; inversion ratio 0.59
WORKLOAD cachyos-EEVDF-xz-compression
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
long-range-dependent dispatch-rate hurst 0.737602 MED
discord 1081ms profile_vs_mean 1.53648 MED
long-range-dependent migration-rate hurst 0.533363 MED
ioctl-slow 131339/xz duration_ms 0.00061 LOW
hot-cpu 7 l2_miss_share 7.61194 LOW
ioctl-slow 131339/xz duration_ms 0.0005 LOW
CAPTURE complete -- 78551 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 15.9k wake2run; p50 6us p99 1633us p999 3078us worst 4869us; 93.4% fast(<100us) / 2.2% mid / 4.5% tick-floor(>=900us); 0.1% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~14188 distinct values; inversion ratio 0.48
WORKLOAD cachyos-PANDEMONIUM-ADAPTIVE-primes
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
hot-cpu 15 l2_miss_share 31.2577 HIGH
long-range-dependent migration-rate hurst 0.611752 MED
migration-avalanche migration-rate runs 33 MED
discord 1280ms profile_vs_mean 1.77616 MED
CAPTURE complete -- 17669 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 12.7k wake2run; p50 4us p99 638us p999 872us worst 940us; 79.2% fast(<100us) / 20.8% mid / 0.0% tick-floor(>=900us); 3.4% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~9728 distinct values; inversion ratio 0.44
WORKLOAD cachyos-PANDEMONIUM-ADAPTIVE-stress-ng-cpu-cache-mem
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
long-range-dependent migration-rate hurst 0.795711 MED
migration-avalanche migration-rate runs 472 MED
discord 8308ms profile_vs_mean 1.67198 MED
long-range-dependent dispatch-rate hurst 0.783475 MED
hot-cpu 2 l2_miss_share 6.78541 LOW
CAPTURE complete -- 132421 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 121k wake2run; p50 4us p99 697us p999 989us worst 2301us; 80.8% fast(<100us) / 18.9% mid / 0.2% tick-floor(>=900us); 3.9% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~94907 distinct values; inversion ratio 0.46
WORKLOAD cachyos-PANDEMONIUM-ADAPTIVE-xz-compression
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
long-range-dependent dispatch-rate hurst 0.685081 MED
migration-avalanche migration-rate runs 99 MED
discord 1912ms profile_vs_mean 1.69136 MED
long-range-dependent migration-rate hurst 0.660835 MED
ioctl-slow 131809/xz duration_ms 0.00052 LOW
hot-cpu 7 l2_miss_share 7.04205 LOW
ioctl-slow 131809/xz duration_ms 0.00039 LOW
CAPTURE complete -- 89601 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 27.4k wake2run; p50 5us p99 701us p999 970us worst 2527us; 81.8% fast(<100us) / 18.0% mid / 0.2% tick-floor(>=900us); 5.4% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~23527 distinct values; inversion ratio 0.50
WORKLOAD cachyos-PANDEMONIUM-BPF-primes
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
hot-cpu 15 l2_miss_share 28.9233 HIGH
long-range-dependent migration-rate hurst 0.608168 MED
migration-avalanche migration-rate runs 60 MED
discord 935ms profile_vs_mean 1.66127 MED
CAPTURE complete -- 17277 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 12.3k wake2run; p50 4us p99 827us p999 999us worst 1005us; 77.6% fast(<100us) / 21.9% mid / 0.6% tick-floor(>=900us); 6.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~10976 distinct values; inversion ratio 0.43
WORKLOAD cachyos-PANDEMONIUM-BPF-stress-ng-cpu-cache-mem
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
long-range-dependent dispatch-rate hurst 0.820745 MED
migration-avalanche migration-rate runs 773 MED
discord 4647ms profile_vs_mean 1.56829 MED
long-range-dependent migration-rate hurst 0.802065 MED
hot-cpu 2 l2_miss_share 7.35471 LOW
CAPTURE complete -- 129238 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 117k wake2run; p50 4us p99 767us p999 999us worst 1661us; 78.7% fast(<100us) / 20.9% mid / 0.5% tick-floor(>=900us); 6.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~100949 distinct values; inversion ratio 0.54
WORKLOAD cachyos-PANDEMONIUM-BPF-xz-compression
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
long-range-dependent dispatch-rate hurst 0.706462 MED
migration-avalanche migration-rate runs 124 MED
discord 1281ms profile_vs_mean 1.61585 MED
long-range-dependent migration-rate hurst 0.673024 MED
ioctl-slow 131571/xz duration_ms 0.00049 LOW
hot-cpu 13 l2_miss_share 6.96319 LOW
ioctl-slow 131571/xz duration_ms 0.00039 LOW
CAPTURE complete -- 87418 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 25.2k wake2run; p50 7us p99 721us p999 999us worst 1537us; 77.8% fast(<100us) / 21.7% mid / 0.5% tick-floor(>=900us); 5.7% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~24404 distinct values; inversion ratio 0.46
WORKLOAD cachyos-scx_beerland-primes
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
hot-cpu 15 l2_miss_share 33.4202 HIGH
long-range-dependent migration-rate hurst 0.621056 MED
migration-avalanche migration-rate runs 97 MED
discord 1039ms profile_vs_mean 1.80183 MED
long-range-dependent dispatch-rate hurst 0.592524 MED
CAPTURE complete -- 14969 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 10.0k wake2run; p50 4us p99 1001us p999 2000us worst 7562us; 82.3% fast(<100us) / 14.1% mid / 3.6% tick-floor(>=900us); 14.1% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~8452 distinct values; inversion ratio 0.49
WORKLOAD cachyos-scx_beerland-stress-ng-cpu-cache-mem
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
kthread-strand comm#e1c397f1 max_strand_ms 16.9059 MED
long-range-dependent dispatch-rate hurst 0.899638 MED
migration-avalanche migration-rate runs 1844 MED
kthread-strand comm#dca24885 max_strand_ms 14.7815 MED
kthread-strand comm#ff850676 max_strand_ms 11.9893 MED
long-range-dependent migration-rate hurst 0.855297 MED
hot-cpu 15 l2_miss_share 6.82724 LOW
CAPTURE complete -- 109369 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 93.3k wake2run; p50 4us p99 1001us p999 2000us worst 19004us; 82.5% fast(<100us) / 14.0% mid / 3.5% tick-floor(>=900us); 18.4% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~88953 distinct values; inversion ratio 0.37
REPORT kstrand
VERDICT: 3 strands across 3 per-CPU kthreads; worst HELD strand 16.9ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#e1c397f1 8 1 16.9 16.9 1 0 tid=0 0%
comm#dca24885 9 1 14.8 14.8 1 0 tid=0 0%
comm#ff850676 2 1 12.0 12.0 1 0 tid=0 0%
WORKLOAD cachyos-scx_beerland-xz-compression
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
long-range-dependent dispatch-rate hurst 0.76934 MED
migration-avalanche migration-rate runs 235 MED
discord 1933ms profile_vs_mean 1.72298 MED
long-range-dependent migration-rate hurst 0.702463 MED
ioctl-slow 134347/xz duration_ms 0.0006 LOW
hot-cpu 12 l2_miss_share 7.59787 LOW
ioctl-slow 134347/xz duration_ms 0.00037 LOW
CAPTURE complete -- 77135 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 14.9k wake2run; p50 8us p99 1001us p999 2004us worst 4653us; 78.4% fast(<100us) / 16.3% mid / 5.3% tick-floor(>=900us); 17.6% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~13761 distinct values; inversion ratio 0.53
WORKLOAD cachyos-scx_bpfland-primes
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
hot-cpu 15 l2_miss_share 35.6775 HIGH
migration-avalanche migration-rate runs 403 MED
CAPTURE complete -- 9927 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 5.0k wake2run; p50 8us p99 1999us p999 2007us worst 3002us; 84.7% fast(<100us) / 2.3% mid / 13.1% tick-floor(>=900us); 39.1% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~4738 distinct values; inversion ratio 0.49
WORKLOAD cachyos-scx_bpfland-stress-ng-cpu-cache-mem
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
kthread-strand comm#cfa272d9 max_strand_ms 36 MED
long-range-dependent dispatch-rate hurst 0.831714 MED
migration-avalanche migration-rate runs 2583 MED
discord 9354ms profile_vs_mean 1.59875 MED
kthread-strand comm#eb6af029 max_strand_ms 33.998 MED
kthread-strand comm#dca24885 max_strand_ms 8.99823 MED
kthread-strand comm#fa4c2d9c max_strand_ms 8.00238 MED
kthread-strand comm#0dbd544b max_strand_ms 6.00218 MED
kthread-strand comm#12f62659 max_strand_ms 6.00131 MED
kthread-strand comm#288f4239 max_strand_ms 5.99398 MED
kthread-strand comm#da1fa6ff max_strand_ms 5.00063 MED
kthread-strand comm#0357d927 max_strand_ms 5.0003 MED
kthread-strand comm#658ebbd8 max_strand_ms 5.00007 MED
long-range-dependent migration-rate hurst 0.742914 MED
hot-cpu 3 l2_miss_share 6.65755 LOW
CAPTURE complete -- 76749 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 59.5k wake2run; p50 6us p99 1163us p999 2007us worst 2015us; 86.6% fast(<100us) / 2.2% mid / 11.2% tick-floor(>=900us); 35.9% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~56738 distinct values; inversion ratio 0.38
REPORT kstrand
VERDICT: 27 strands across 10 per-CPU kthreads; worst HELD strand 36.0ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#cfa272d9 15 9 36.0 36.0 9 0 tid=0 0%
comm#eb6af029 15 10 34.0 34.0 10 0 tid=0 0%
comm#dca24885 9 1 9.0 9.0 1 0 tid=0 0%
comm#fa4c2d9c 3 1 8.0 8.0 1 0 tid=0 0%
comm#0dbd544b 7 1 6.0 6.0 1 0 tid=0 0%
comm#12f62659 6 1 6.0 6.0 1 0 tid=0 0%
comm#288f4239 0 1 6.0 6.0 1 0 tid=0 0%
comm#da1fa6ff 13 1 5.0 5.0 1 0 tid=0 0%
comm#0357d927 1 1 5.0 5.0 1 0 tid=0 0%
comm#658ebbd8 0 1 5.0 5.0 1 0 tid=0 0%
WORKLOAD cachyos-scx_bpfland-xz-compression
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
long-range-dependent migration-rate hurst 0.591574 MED
migration-avalanche migration-rate runs 692 MED
discord 946ms profile_vs_mean 1.50915 MED
ioctl-slow 132043/xz duration_ms 0.00068 LOW
hot-cpu 3 l2_miss_share 6.9744 LOW
ioctl-slow 132043/xz duration_ms 0.0005 LOW
CAPTURE complete -- 71905 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 9.7k wake2run; p50 12us p99 1110us p999 2005us worst 2020us; 75.3% fast(<100us) / 7.4% mid / 17.3% tick-floor(>=900us); 36.9% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~9667 distinct values; inversion ratio 0.54
WORKLOAD cachyos-scx_cake-primes
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
hot-cpu 15 l2_miss_share 33.0111 HIGH
kthread-strand comm#1beaf0af max_strand_ms 15.0001 MED
long-range-dependent migration-rate hurst 0.577591 MED
CAPTURE complete -- 16833 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 11.9k wake2run; p50 3us p99 1001us p999 2000us worst 2998us; 91.4% fast(<100us) / 1.6% mid / 7.1% tick-floor(>=900us); 1.1% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~8527 distinct values; inversion ratio 0.45
REPORT kstrand
VERDICT: 1 strands across 1 per-CPU kthreads; worst HELD strand 15.0ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#1beaf0af 4 1 15.0 15.0 1 0 tid=0 0%
WORKLOAD cachyos-scx_cake-stress-ng-cpu-cache-mem
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
kthread-strand comm#1beaf0af max_strand_ms 17.9999 MED
long-range-dependent dispatch-rate hurst 0.845075 MED
migration-avalanche migration-rate runs 62 MED
discord 6180ms profile_vs_mean 1.65239 MED
kthread-strand comm#ff850676 max_strand_ms 12.9979 MED
kthread-strand comm#658ebbd8 max_strand_ms 11.009 MED
long-range-dependent migration-rate hurst 0.758039 MED
hot-cpu 15 l2_miss_share 6.87624 LOW
CAPTURE complete -- 132732 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 117k wake2run; p50 3us p99 1001us p999 2000us worst 2274us; 92.5% fast(<100us) / 1.4% mid / 6.1% tick-floor(>=900us); 1.2% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~89285 distinct values; inversion ratio 0.52
REPORT kstrand
VERDICT: 4 strands across 3 per-CPU kthreads; worst HELD strand 18.0ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#1beaf0af 4 1 18.0 18.0 1 0 tid=0 0%
comm#ff850676 2 1 13.0 13.0 1 0 tid=0 0%
comm#658ebbd8 0 2 11.0 11.0 2 0 tid=0 0%
WORKLOAD cachyos-scx_cake-xz-compression
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
long-range-dependent dispatch-rate hurst 0.713483 MED
migration-avalanche migration-rate runs 6 MED
long-range-dependent migration-rate hurst 0.613489 MED
ioctl-slow 133877/xz duration_ms 0.00068 LOW
hot-cpu 13 l2_miss_share 7.57882 LOW
ioctl-slow 133877/xz duration_ms 0.00037 LOW
CAPTURE complete -- 85131 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 22.9k wake2run; p50 4us p99 1002us p999 2002us worst 3000us; 91.8% fast(<100us) / 1.6% mid / 6.6% tick-floor(>=900us); 1.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~17927 distinct values; inversion ratio 0.46
WORKLOAD cachyos-scx_cosmos-primes
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
hot-cpu 15 l2_miss_share 30.48 HIGH
kthread-strand comm#c43b8240 max_strand_ms 11.0018 MED
mean-reverting dispatch-rate hurst 0.422536 MED
migration-avalanche migration-rate runs 554 MED
kthread-strand comm#ca8154b4 max_strand_ms 11.0005 MED
kthread-strand comm#12f62659 max_strand_ms 6.99387 MED
kthread-strand comm#eb6af542 max_strand_ms 5.99421 MED
kthread-strand comm#ff850676 max_strand_ms 5.00154 MED
kthread-strand comm#dca24885 max_strand_ms 5.00044 MED
CAPTURE complete -- 10723 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 5.8k wake2run; p50 8us p99 2000us p999 2002us worst 2003us; 84.5% fast(<100us) / 2.0% mid / 13.5% tick-floor(>=900us); 43.1% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~5673 distinct values; inversion ratio 0.53
REPORT kstrand
VERDICT: 8 strands across 6 per-CPU kthreads; worst HELD strand 11.0ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#c43b8240 12 2 11.0 11.0 2 0 tid=0 0%
comm#ca8154b4 14 2 11.0 11.0 2 0 tid=0 0%
comm#12f62659 6 1 7.0 7.0 1 0 tid=0 0%
comm#eb6af542 10 1 6.0 6.0 1 0 tid=0 0%
comm#ff850676 2 1 5.0 5.0 1 0 tid=0 0%
comm#dca24885 9 1 5.0 5.0 1 0 tid=0 0%
WORKLOAD cachyos-scx_cosmos-stress-ng-cpu-cache-mem
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
kthread-strand comm#cfa26f73 max_strand_ms 80.9713 MED
long-range-dependent dispatch-rate hurst 0.857558 MED
migration-avalanche migration-rate runs 3860 MED
kthread-strand comm#fa4c2883 max_strand_ms 65.0103 MED
kthread-strand comm#180cd039 max_strand_ms 59.0699 MED
kthread-strand comm#eb6af029 max_strand_ms 57.9982 MED
kthread-strand comm#658ebbd8 max_strand_ms 57.0072 MED
kthread-strand comm#ff850676 max_strand_ms 53.0113 MED
kthread-strand comm#ca8154b4 max_strand_ms 43.9727 MED
kthread-strand comm#1beaf0af max_strand_ms 40.9893 MED
kthread-strand comm#0dbd57b1 max_strand_ms 40.9873 MED
kthread-strand comm#da1fa6ff max_strand_ms 40.9757 MED
kthread-strand comm#0357d927 max_strand_ms 33.0007 MED
kthread-strand comm#493ffe92 max_strand_ms 31.056 MED
kthread-strand comm#c43b8240 max_strand_ms 13.0136 MED
kthread-strand comm#dca24885 max_strand_ms 11.0089 MED
kthread-strand comm#e3146723 max_strand_ms 10.9954 MED
kthread-strand comm#12f62659 max_strand_ms 7.98428 MED
kthread-strand comm#fa4c2d9c max_strand_ms 7.00747 MED
kthread-strand comm#288f4239 max_strand_ms 6.07379 MED
kthread-strand comm#eb6af542 max_strand_ms 5.99442 MED
kthread-strand comm#eb6af38f max_strand_ms 5.99231 MED
kthread-strand comm#e1c397f1 max_strand_ms 5.97846 MED
kthread-strand comm#65897344 max_strand_ms 5.06718 MED
long-range-dependent migration-rate hurst 0.7532 MED
hot-cpu 15 l2_miss_share 6.69806 LOW
CAPTURE complete -- 87204 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 69.5k wake2run; p50 6us p99 1999us p999 2002us worst 3002us; 88.7% fast(<100us) / 1.7% mid / 9.5% tick-floor(>=900us); 39.1% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~69519 distinct values; inversion ratio 0.37
REPORT kstrand
VERDICT: 115 strands across 22 per-CPU kthreads; worst HELD strand 81.0ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#cfa26f73 15 5 81.0 81.0 5 0 tid=0 0%
comm#fa4c2883 3 7 65.0 65.0 7 0 tid=0 0%
comm#180cd039 5 2 59.1 59.1 2 0 tid=0 0%
comm#eb6af029 15 8 58.0 58.0 8 0 tid=0 0%
comm#658ebbd8 0 26 57.0 57.0 26 0 tid=0 0%
comm#ff850676 2 8 53.0 53.0 8 0 tid=0 0%
comm#ca8154b4 14 9 44.0 44.0 9 0 tid=0 0%
comm#1beaf0af 4 7 41.0 41.0 7 0 tid=0 0%
comm#0dbd57b1 7 7 41.0 41.0 7 0 tid=0 0%
comm#da1fa6ff 13 4 41.0 41.0 4 0 tid=0 0%
comm#0357d927 1 10 33.0 33.0 10 0 tid=0 0%
comm#493ffe92 10 5 31.1 31.1 5 0 tid=0 0%
comm#c43b8240 12 3 13.0 13.0 3 0 tid=0 0%
comm#dca24885 9 3 11.0 11.0 3 0 tid=0 0%
comm#e3146723 11 4 11.0 11.0 4 0 tid=0 0%
comm#12f62659 6 1 8.0 8.0 1 0 tid=0 0%
comm#fa4c2d9c 3 1 7.0 7.0 1 0 tid=0 0%
comm#288f4239 0 1 6.1 6.1 1 0 tid=0 0%
comm#eb6af542 10 1 6.0 6.0 1 0 tid=0 0%
comm#eb6af38f 13 1 6.0 6.0 1 0 tid=0 0%
WORKLOAD cachyos-scx_cosmos-xz-compression
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
kthread-strand comm#ff850676 max_strand_ms 15.9989 MED
migration-avalanche migration-rate runs 1115 MED
discord 1475ms profile_vs_mean 1.52145 MED
kthread-strand comm#cfa26dc0 max_strand_ms 6.00018 MED
kthread-strand comm#fa4c2883 max_strand_ms 5.98748 MED
kthread-strand comm#cfa26f73 max_strand_ms 5.98713 MED
kthread-strand comm#e3146723 max_strand_ms 5.01236 MED
kthread-strand comm#0357d927 max_strand_ms 5.00584 MED
ioctl-slow 133650/xz duration_ms 0.00065 LOW
hot-cpu 10 l2_miss_share 6.91419 LOW
ioctl-slow 133650/xz duration_ms 0.00056 LOW
CAPTURE complete -- 74639 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 12.4k wake2run; p50 10us p99 1014us p999 1101us worst 2004us; 83.7% fast(<100us) / 4.2% mid / 12.1% tick-floor(>=900us); 39.7% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~12058 distinct values; inversion ratio 0.49
REPORT kstrand
VERDICT: 6 strands across 6 per-CPU kthreads; worst HELD strand 16.0ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#ff850676 2 1 16.0 16.0 1 0 tid=0 0%
comm#cfa26dc0 15 1 6.0 6.0 1 0 tid=0 0%
comm#fa4c2883 3 1 6.0 6.0 1 0 tid=0 0%
comm#cfa26f73 15 1 6.0 6.0 1 0 tid=0 0%
comm#e3146723 11 1 5.0 5.0 1 0 tid=0 0%
comm#0357d927 1 1 5.0 5.0 1 0 tid=0 0%
WORKLOAD cachyos-scx_flash-primes
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
hot-cpu 15 l2_miss_share 32.2277 HIGH
kthread-strand comm#ff850676 max_strand_ms 31.4078 MED
mean-reverting dispatch-rate hurst 0.445929 MED
migration-avalanche migration-rate runs 615 MED
kthread-strand comm#dca24885 max_strand_ms 10.4851 MED
CAPTURE complete -- 12492 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 7.5k wake2run; p50 7us p99 1008us p999 1011us worst 2003us; 85.6% fast(<100us) / 1.4% mid / 13.0% tick-floor(>=900us); 40.7% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~7418 distinct values; inversion ratio 0.48
REPORT kstrand
VERDICT: 2 strands across 2 per-CPU kthreads; worst HELD strand 31.4ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#ff850676 2 1 31.4 31.4 1 0 tid=0 0%
comm#dca24885 9 1 10.5 10.5 1 0 tid=0 0%
WORKLOAD cachyos-scx_flash-stress-ng-cpu-cache-mem
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
kthread-strand comm#180cd039 max_strand_ms 55.3122 MED
long-range-dependent dispatch-rate hurst 0.83522 MED
migration-avalanche migration-rate runs 4522 MED
discord 3098ms profile_vs_mean 1.86917 MED
kthread-strand comm#0dbd57b1 max_strand_ms 48.16 MED
kthread-strand comm#cfa26f73 max_strand_ms 11.9715 MED
kthread-strand comm#eb6af029 max_strand_ms 8.99901 MED
long-range-dependent migration-rate hurst 0.740261 MED
hot-cpu 15 l2_miss_share 6.84854 LOW
CAPTURE complete -- 96211 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 79.1k wake2run; p50 6us p99 1007us p999 1010us worst 1574us; 89.1% fast(<100us) / 1.3% mid / 9.6% tick-floor(>=900us); 38.5% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~77863 distinct values; inversion ratio 0.38
REPORT kstrand
VERDICT: 9 strands across 4 per-CPU kthreads; worst HELD strand 55.3ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#180cd039 5 4 55.3 55.3 4 0 tid=0 0%
comm#0dbd57b1 7 1 48.2 48.2 1 0 tid=0 0%
comm#cfa26f73 15 1 12.0 12.0 1 0 tid=0 0%
comm#eb6af029 15 3 9.0 9.0 3 0 tid=0 0%
WORKLOAD cachyos-scx_flash-xz-compression
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
kthread-strand comm#cfa26f73 max_strand_ms 164 HIGH
migration-avalanche migration-rate runs 980 MED
kthread-strand comm#da1fa6ff max_strand_ms 58.2776 MED
kthread-strand comm#fa4c2883 max_strand_ms 22.0078 MED
kthread-strand comm#eb6af029 max_strand_ms 5.1206 MED
ioctl-slow 134110/xz duration_ms 0.00071 LOW
hot-cpu 1 l2_miss_share 6.86806 LOW
ioctl-slow 134110/xz duration_ms 0.00035 LOW
CAPTURE complete -- 77462 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 15.2k wake2run; p50 9us p99 1008us p999 1012us worst 2003us; 85.8% fast(<100us) / 2.8% mid / 11.4% tick-floor(>=900us); 41.7% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~14526 distinct values; inversion ratio 0.47
REPORT kstrand
VERDICT: 7 strands across 4 per-CPU kthreads; worst HELD strand 164.0ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#cfa26f73 15 2 164.0 164.0 2 0 tid=0 0%
comm#da1fa6ff 13 3 58.3 58.3 3 0 tid=0 0%
comm#fa4c2883 3 1 22.0 22.0 1 0 tid=0 0%
comm#eb6af029 15 1 5.1 5.1 1 0 tid=0 0%
WORKLOAD cachyos-scx_flow-primes
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
hot-cpu 15 l2_miss_share 34.0987 HIGH
long-range-dependent migration-rate hurst 0.571352 MED
migration-avalanche migration-rate runs 80 MED
discord 576ms profile_vs_mean 1.57023 MED
CAPTURE complete -- 14828 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 9.9k wake2run; p50 4us p99 1002us p999 1004us worst 1093us; 89.8% fast(<100us) / 2.0% mid / 8.1% tick-floor(>=900us); 5.8% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~8161 distinct values; inversion ratio 0.37
WORKLOAD cachyos-scx_flow-stress-ng-cpu-cache-mem
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
long-range-dependent dispatch-rate hurst 0.878818 MED
migration-avalanche migration-rate runs 854 MED
discord 7606ms profile_vs_mean 1.79166 MED
long-range-dependent migration-rate hurst 0.680626 MED
hot-cpu 15 l2_miss_share 7.11134 LOW
CAPTURE complete -- 116515 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 99.8k wake2run; p50 3us p99 1001us p999 1004us worst 2004us; 90.5% fast(<100us) / 2.7% mid / 6.8% tick-floor(>=900us); 5.4% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~90439 distinct values; inversion ratio 0.41
WORKLOAD cachyos-scx_flow-xz-compression
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
long-range-dependent dispatch-rate hurst 0.752863 MED
migration-avalanche migration-rate runs 119 MED
discord 1787ms profile_vs_mean 1.72329 MED
long-range-dependent migration-rate hurst 0.630685 MED
ioctl-slow 132742/xz duration_ms 0.00055 LOW
hot-cpu 10 l2_miss_share 7.6368 LOW
ioctl-slow 132742/xz duration_ms 0.00039 LOW
CAPTURE complete -- 78746 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 16.5k wake2run; p50 6us p99 1000us p999 1004us worst 2000us; 86.9% fast(<100us) / 4.6% mid / 8.4% tick-floor(>=900us); 5.8% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~15248 distinct values; inversion ratio 0.62
WORKLOAD cachyos-scx_lavd-primes
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
hot-cpu 15 l2_miss_share 30.1681 HIGH
kthread-strand comm#0357d927 max_strand_ms 11.0015 MED
long-range-dependent migration-rate hurst 0.599899 MED
migration-avalanche migration-rate runs 73 MED
discord 1034ms profile_vs_mean 1.5432 MED
kthread-strand comm#658ebbd8 max_strand_ms 11.0014 MED
kthread-strand comm#12f62659 max_strand_ms 7.00041 MED
kthread-strand comm#180cd039 max_strand_ms 5.99973 MED
kthread-strand comm#288f3d20 max_strand_ms 5.99195 MED
kthread-strand comm#e3146723 max_strand_ms 5.98212 MED
kthread-strand comm#ae46779d max_strand_ms 5.95852 MED
kthread-strand comm#cfa26f73 max_strand_ms 5.9083 MED
kthread-strand comm#cfa272d9 max_strand_ms 5.90021 MED
kthread-strand comm#0dbd57b1 max_strand_ms 5.87918 MED
kthread-strand comm#cfa27126 max_strand_ms 5.22103 MED
kthread-strand comm#2f099b5a max_strand_ms 5.16111 MED
kthread-strand comm#fa4c2d9c max_strand_ms 5.00168 MED
kthread-strand comm#dca24885 max_strand_ms 5.00017 MED
mean-reverting dispatch-rate hurst 0.433764 MED
CAPTURE complete -- 8315 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 3.3k wake2run; p50 11us p99 3071us p999 6005us worst 11999us; 77.1% fast(<100us) / 5.5% mid / 17.3% tick-floor(>=900us); 15.5% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~3267 distinct values; inversion ratio 0.50
REPORT kstrand
VERDICT: 19 strands across 14 per-CPU kthreads; worst HELD strand 11.0ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#0357d927 1 3 11.0 11.0 3 0 tid=0 0%
comm#658ebbd8 0 2 11.0 11.0 2 0 tid=0 0%
comm#12f62659 6 2 7.0 7.0 2 0 tid=0 0%
comm#180cd039 5 1 6.0 6.0 1 0 tid=0 0%
comm#288f3d20 3 1 6.0 6.0 1 0 tid=0 0%
comm#e3146723 11 1 6.0 6.0 1 0 tid=0 0%
comm#ae46779d 1 2 6.0 6.0 2 0 tid=0 0%
comm#cfa26f73 15 1 5.9 5.9 1 0 tid=0 0%
comm#cfa272d9 15 1 5.9 5.9 1 0 tid=0 0%
comm#0dbd57b1 7 1 5.9 5.9 1 0 tid=0 0%
comm#cfa27126 15 1 5.2 5.2 1 0 tid=0 0%
comm#2f099b5a 2 1 5.2 5.2 1 0 tid=0 0%
comm#fa4c2d9c 3 1 5.0 5.0 1 0 tid=0 0%
comm#dca24885 9 1 5.0 5.0 1 0 tid=0 0%
WORKLOAD cachyos-scx_lavd-stress-ng-cpu-cache-mem
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
kthread-strand comm#658ebbd8 max_strand_ms 41.004 MED
long-range-dependent dispatch-rate hurst 0.822676 MED
migration-avalanche migration-rate runs 1218 MED
discord 2183ms profile_vs_mean 1.68272 MED
kthread-strand comm#0357d927 max_strand_ms 16.9997 MED
kthread-strand comm#fa4c2d9c max_strand_ms 15.0009 MED
kthread-strand comm#ff850b8f max_strand_ms 12.0719 MED
kthread-strand comm#0dbd57b1 max_strand_ms 11.0011 MED
kthread-strand comm#180cd039 max_strand_ms 10.9934 MED
kthread-strand comm#12f62659 max_strand_ms 9.98566 MED
kthread-strand comm#1beaf0af max_strand_ms 9.0921 MED
kthread-strand comm#cfa26f73 max_strand_ms 9.00174 MED
kthread-strand comm#2f099b5a max_strand_ms 7.00047 MED
kthread-strand comm#ca8154b4 max_strand_ms 6.0009 MED
kthread-strand comm#da1fa6ff max_strand_ms 5.98388 MED
kthread-strand comm#dca24885 max_strand_ms 5.98233 MED
kthread-strand comm#e3146723 max_strand_ms 5.00467 MED
kthread-strand comm#e1c397f1 max_strand_ms 5.00061 MED
kthread-strand comm#c43b8240 max_strand_ms 5.00002 MED
long-range-dependent migration-rate hurst 0.60285 MED
hot-cpu 15 l2_miss_share 6.85061 LOW
CAPTURE complete -- 46335 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 34.9k wake2run; p50 9us p99 3004us p999 5005us worst 9515us; 80.7% fast(<100us) / 5.0% mid / 14.3% tick-floor(>=900us); 16.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~34867 distinct values; inversion ratio 0.33
REPORT kstrand
VERDICT: 104 strands across 16 per-CPU kthreads; worst HELD strand 41.0ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#658ebbd8 0 40 41.0 41.0 40 0 tid=0 0%
comm#0357d927 1 5 17.0 17.0 5 0 tid=0 0%
comm#fa4c2d9c 3 6 15.0 15.0 6 0 tid=0 0%
comm#ff850b8f 2 15 12.1 12.1 15 0 tid=0 0%
comm#0dbd57b1 7 4 11.0 11.0 4 0 tid=0 0%
comm#180cd039 5 6 11.0 11.0 6 0 tid=0 0%
comm#12f62659 6 4 10.0 10.0 4 0 tid=0 0%
comm#1beaf0af 4 5 9.1 9.1 5 0 tid=0 0%
comm#cfa26f73 15 8 9.0 9.0 8 0 tid=0 0%
comm#2f099b5a 2 1 7.0 7.0 1 0 tid=0 0%
comm#ca8154b4 14 4 6.0 6.0 4 0 tid=0 0%
comm#da1fa6ff 13 1 6.0 6.0 1 0 tid=0 0%
comm#dca24885 9 2 6.0 6.0 2 0 tid=0 0%
comm#e3146723 11 1 5.0 5.0 1 0 tid=0 0%
comm#e1c397f1 8 1 5.0 5.0 1 0 tid=0 0%
comm#c43b8240 12 1 5.0 5.0 1 0 tid=0 0%
WORKLOAD cachyos-scx_lavd-xz-compression
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
kthread-strand comm#658ebbd8 max_strand_ms 20.0027 MED
long-range-dependent migration-rate hurst 0.584244 MED
migration-avalanche migration-rate runs 160 MED
kthread-strand comm#cfa272d9 max_strand_ms 10.9943 MED
kthread-strand comm#fa4c2d9c max_strand_ms 6.00169 MED
kthread-strand comm#ca8154b4 max_strand_ms 6.00062 MED
kthread-strand comm#1beaf0af max_strand_ms 5.99899 MED
kthread-strand comm#12f62659 max_strand_ms 5.99308 MED
kthread-strand comm#ff850b8f max_strand_ms 5.98685 MED
kthread-strand comm#e1c397f1 max_strand_ms 5.96879 MED
kthread-strand comm#e3146723 max_strand_ms 5.96029 MED
kthread-strand comm#da1fa6ff max_strand_ms 5.95419 MED
kthread-strand comm#0357d927 max_strand_ms 5.17002 MED
kthread-strand comm#0dbd57b1 max_strand_ms 5.00005 MED
ioctl-slow 132508/xz duration_ms 0.00065 LOW
hot-cpu 9 l2_miss_share 7.55792 LOW
ioctl-slow 132508/xz duration_ms 0.00048 LOW
CAPTURE complete -- 68831 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 6.6k wake2run; p50 16us p99 3005us p999 4050us worst 5046us; 71.4% fast(<100us) / 11.4% mid / 17.2% tick-floor(>=900us); 15.7% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~6627 distinct values; inversion ratio 0.56
REPORT kstrand
VERDICT: 26 strands across 12 per-CPU kthreads; worst HELD strand 20.0ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#658ebbd8 0 8 20.0 20.0 8 0 tid=0 0%
comm#cfa272d9 15 5 11.0 11.0 5 0 tid=0 0%
comm#fa4c2d9c 3 2 6.0 6.0 2 0 tid=0 0%
comm#ca8154b4 14 1 6.0 6.0 1 0 tid=0 0%
comm#1beaf0af 4 1 6.0 6.0 1 0 tid=0 0%
comm#12f62659 6 2 6.0 6.0 2 0 tid=0 0%
comm#ff850b8f 2 1 6.0 6.0 1 0 tid=0 0%
comm#e1c397f1 8 1 6.0 6.0 1 0 tid=0 0%
comm#e3146723 11 1 6.0 6.0 1 0 tid=0 0%
comm#da1fa6ff 13 1 6.0 6.0 1 0 tid=0 0%
comm#0357d927 1 2 5.2 5.2 2 0 tid=0 0%
comm#0dbd57b1 7 1 5.0 5.0 1 0 tid=0 0%
WORKLOAD cachyos-scx_p2dq-primes
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
hot-cpu 15 l2_miss_share 33.3148 HIGH
kthread-strand comm#658ebbd8 max_strand_ms 20.9985 MED
long-range-dependent migration-rate hurst 0.547073 MED
kthread-strand comm#2f099b5a max_strand_ms 19.9997 MED
kthread-strand comm#ff850b8f max_strand_ms 19.9997 MED
kthread-strand comm#e3146723 max_strand_ms 19.999 MED
kthread-strand comm#cfa26f73 max_strand_ms 19.9955 MED
kthread-strand comm#0357d927 max_strand_ms 14.9922 MED
kthread-strand comm#0dbd57b1 max_strand_ms 13.0008 MED
kthread-strand comm#dca24885 max_strand_ms 13.0006 MED
kthread-strand comm#12f62659 max_strand_ms 12.9993 MED
kthread-strand comm#493ffe92 max_strand_ms 12.9975 MED
kthread-strand comm#fa4c2883 max_strand_ms 10.0013 MED
CAPTURE complete -- 9788 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 4.8k wake2run; p50 6us p99 2001us p999 4000us worst 9001us; 82.1% fast(<100us) / 2.1% mid / 15.8% tick-floor(>=900us); 1.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~4500 distinct values; inversion ratio 0.46
REPORT kstrand
VERDICT: 24 strands across 11 per-CPU kthreads; worst HELD strand 21.0ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#658ebbd8 0 9 21.0 21.0 9 0 tid=0 0%
comm#2f099b5a 2 1 20.0 20.0 1 0 tid=0 0%
comm#ff850b8f 2 3 20.0 20.0 3 0 tid=0 0%
comm#e3146723 11 2 20.0 20.0 2 0 tid=0 0%
comm#cfa26f73 15 3 20.0 20.0 3 0 tid=0 0%
comm#0357d927 1 1 15.0 15.0 1 0 tid=0 0%
comm#0dbd57b1 7 1 13.0 13.0 1 0 tid=0 0%
comm#dca24885 9 1 13.0 13.0 1 0 tid=0 0%
comm#12f62659 6 1 13.0 13.0 1 0 tid=0 0%
comm#493ffe92 10 1 13.0 13.0 1 0 tid=0 0%
comm#fa4c2883 3 1 10.0 10.0 1 0 tid=0 0%
WORKLOAD cachyos-scx_p2dq-stress-ng-cpu-cache-mem
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
kthread-strand comm#cfa272d9 max_strand_ms 20.996 MED
long-range-dependent dispatch-rate hurst 0.881665 MED
migration-avalanche migration-rate runs 81 MED
kthread-strand comm#12f62659 max_strand_ms 20.9924 MED
kthread-strand comm#658ebbd8 max_strand_ms 20.0005 MED
kthread-strand comm#dca24885 max_strand_ms 20.0002 MED
kthread-strand comm#180cd039 max_strand_ms 20.0001 MED
kthread-strand comm#1beaf0af max_strand_ms 20.0001 MED
kthread-strand comm#0dbd57b1 max_strand_ms 19.9999 MED
kthread-strand comm#ff850b8f max_strand_ms 19.9999 MED
kthread-strand comm#da1fa6ff max_strand_ms 19.999 MED
kthread-strand comm#ca8154b4 max_strand_ms 19.9986 MED
kthread-strand comm#0357d927 max_strand_ms 19.9934 MED
kthread-strand comm#65897344 max_strand_ms 19.001 MED
kthread-strand comm#fa4c2883 max_strand_ms 19.0007 MED
kthread-strand comm#e3146723 max_strand_ms 18.9995 MED
kthread-strand comm#c43b8240 max_strand_ms 18 MED
kthread-strand comm#2f099b5a max_strand_ms 13.9998 MED
kthread-strand comm#d1030141 max_strand_ms 13.9995 MED
kthread-strand comm#e1c397f1 max_strand_ms 13.9994 MED
kthread-strand comm#0dbd544b max_strand_ms 13.0061 MED
kthread-strand comm#493ffe92 max_strand_ms 11.984 MED
long-range-dependent migration-rate hurst 0.734272 MED
hot-cpu 3 l2_miss_share 6.79402 LOW
CAPTURE complete -- 82061 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 67.0k wake2run; p50 5us p99 1003us p999 2000us worst 10001us; 87.6% fast(<100us) / 1.6% mid / 10.8% tick-floor(>=900us); 0.9% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~65989 distinct values; inversion ratio 0.32
REPORT kstrand
VERDICT: 228 strands across 20 per-CPU kthreads; worst HELD strand 21.0ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#cfa272d9 15 16 21.0 21.0 16 0 tid=0 0%
comm#12f62659 6 19 21.0 21.0 19 0 tid=0 0%
comm#658ebbd8 0 44 20.0 20.0 44 0 tid=0 0%
comm#dca24885 9 15 20.0 20.0 15 0 tid=0 0%
comm#180cd039 5 21 20.0 20.0 21 0 tid=0 0%
comm#1beaf0af 4 7 20.0 20.0 7 0 tid=0 0%
comm#0dbd57b1 7 26 20.0 20.0 26 0 tid=0 0%
comm#ff850b8f 2 16 20.0 20.0 16 0 tid=0 0%
comm#da1fa6ff 13 5 20.0 20.0 5 0 tid=0 0%
comm#ca8154b4 14 7 20.0 20.0 7 0 tid=0 0%
comm#0357d927 1 10 20.0 20.0 10 0 tid=0 0%
comm#65897344 0 3 19.0 19.0 3 0 tid=0 0%
comm#fa4c2883 3 16 19.0 19.0 16 0 tid=0 0%
comm#e3146723 11 7 19.0 19.0 7 0 tid=0 0%
comm#c43b8240 12 4 18.0 18.0 4 0 tid=0 0%
comm#2f099b5a 2 2 14.0 14.0 2 0 tid=0 0%
comm#d1030141 15 1 14.0 14.0 1 0 tid=0 0%
comm#e1c397f1 8 6 14.0 14.0 6 0 tid=0 0%
comm#0dbd544b 7 1 13.0 13.0 1 0 tid=0 0%
comm#493ffe92 10 2 12.0 12.0 2 0 tid=0 0%
WORKLOAD cachyos-scx_p2dq-xz-compression
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
kthread-strand comm#493ffe92 max_strand_ms 19.9999 MED
hot-waker 133198 wakes_issued 37 MED
long-range-dependent dispatch-rate hurst 0.629839 MED
migration-avalanche migration-rate runs 5 MED
kthread-strand comm#2f099b5a max_strand_ms 19.9993 MED
kthread-strand comm#658ebbd8 max_strand_ms 19.9983 MED
kthread-strand comm#cfa26f73 max_strand_ms 19.9949 MED
kthread-strand comm#0357d927 max_strand_ms 19.001 MED
kthread-strand comm#ca8154b4 max_strand_ms 19.0007 MED
kthread-strand comm#e1c397f1 max_strand_ms 19.0002 MED
kthread-strand comm#ff850b8f max_strand_ms 18.9996 MED
kthread-strand comm#da1fa6ff max_strand_ms 18.9982 MED
kthread-strand comm#0dbd57b1 max_strand_ms 17.9999 MED
kthread-strand comm#180cd039 max_strand_ms 17.9968 MED
kthread-strand comm#dca24885 max_strand_ms 16.001 MED
kthread-strand comm#fa4c2883 max_strand_ms 14.9985 MED
kthread-strand comm#c43b8240 max_strand_ms 13.9992 MED
kthread-strand comm#1beaf0af max_strand_ms 13.0009 MED
kthread-strand comm#12f62659 max_strand_ms 13.0001 MED
kthread-strand comm#e3146723 max_strand_ms 6.0002 MED
long-range-dependent migration-rate hurst 0.544507 MED
ioctl-slow 133198/xz duration_ms 0.0006 LOW
hot-cpu 11 l2_miss_share 7.15877 LOW
ioctl-slow 133198/xz duration_ms 0.00042 LOW
CAPTURE complete -- 72649 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 10.4k wake2run; p50 9us p99 1998us p999 3773us worst 8383us; 82.6% fast(<100us) / 3.0% mid / 14.4% tick-floor(>=900us); 2.3% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~10386 distinct values; inversion ratio 0.53
REPORT kstrand
VERDICT: 69 strands across 17 per-CPU kthreads; worst HELD strand 20.0ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#493ffe92 10 3 20.0 20.0 3 0 tid=0 0%
comm#2f099b5a 2 2 20.0 20.0 2 0 tid=0 0%
comm#658ebbd8 0 11 20.0 20.0 11 0 tid=0 0%
comm#cfa26f73 15 8 20.0 20.0 8 0 tid=0 0%
comm#0357d927 1 3 19.0 19.0 3 0 tid=0 0%
comm#ca8154b4 14 2 19.0 19.0 2 0 tid=0 0%
comm#e1c397f1 8 7 19.0 19.0 7 0 tid=0 0%
comm#ff850b8f 2 6 19.0 19.0 6 0 tid=0 0%
comm#da1fa6ff 13 1 19.0 19.0 1 0 tid=0 0%
comm#0dbd57b1 7 6 18.0 18.0 6 0 tid=0 0%
comm#180cd039 5 2 18.0 18.0 2 0 tid=0 0%
comm#dca24885 9 5 16.0 16.0 5 0 tid=0 0%
comm#fa4c2883 3 5 15.0 15.0 5 0 tid=0 0%
comm#c43b8240 12 1 14.0 14.0 1 0 tid=0 0%
comm#1beaf0af 4 4 13.0 13.0 4 0 tid=0 0%
comm#12f62659 6 2 13.0 13.0 2 0 tid=0 0%
comm#e3146723 11 1 6.0 6.0 1 0 tid=0 0%
WORKLOAD cachyos-scx_rustland-primes
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
discord 3716ms profile_vs_mean 3.07866 HIGH
hot-cpu 15 l2_miss_share 29.3195 HIGH
kthread-strand comm#cfa272d9 max_strand_ms 19.9994 MED
long-range-dependent migration-rate hurst 0.579434 MED
migration-avalanche migration-rate runs 32 MED
kthread-strand comm#d1030141 max_strand_ms 15.0009 MED
CAPTURE complete -- 7015 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 1.9k wake2run; p50 74us p99 7999us p999 18319us worst 29001us; 55.5% fast(<100us) / 27.3% mid / 17.2% tick-floor(>=900us); 18.2% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~1937 distinct values; inversion ratio 0.52
REPORT kstrand
VERDICT: 2 strands across 2 per-CPU kthreads; worst HELD strand 20.0ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#cfa272d9 15 1 20.0 20.0 1 0 tid=0 0%
comm#d1030141 15 1 15.0 15.0 1 0 tid=0 0%
WORKLOAD cachyos-scx_rustland-stress-ng-cpu-cache-mem
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
kthread-strand comm#c43b8240 max_strand_ms 13.9884 MED
long-range-dependent dispatch-rate hurst 0.87804 MED
migration-avalanche migration-rate runs 728 MED
kthread-strand comm#cfa272d9 max_strand_ms 9.00827 MED
long-range-dependent migration-rate hurst 0.843961 MED
hot-cpu 15 l2_miss_share 7.08747 LOW
CAPTURE complete -- 35081 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 22.9k wake2run; p50 28us p99 7936us p999 29002us worst 108002us; 69.4% fast(<100us) / 22.8% mid / 7.8% tick-floor(>=900us); 20.4% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~22944 distinct values; inversion ratio 0.38
LOCATED: 1 structured region(s) NEARLY_SORTED[9%..11%]
REPORT kstrand
VERDICT: 2 strands across 2 per-CPU kthreads; worst HELD strand 14.0ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#c43b8240 12 1 14.0 14.0 1 0 tid=0 0%
comm#cfa272d9 15 1 9.0 9.0 1 0 tid=0 0%
WORKLOAD cachyos-scx_rustland-xz-compression
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
kthread-strand comm#cfa272d9 max_strand_ms 17.0005 MED
long-range-dependent migration-rate hurst 0.547358 MED
migration-avalanche migration-rate runs 94 MED
discord 432ms profile_vs_mean 1.59983 MED
ioctl-slow 132981/xz duration_ms 0.0006 LOW
hot-cpu 10 l2_miss_share 7.1306 LOW
ioctl-slow 132981/xz duration_ms 0.00039 LOW
CAPTURE complete -- 65957 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 3.6k wake2run; p50 58us p99 9002us p999 26998us worst 87001us; 58.7% fast(<100us) / 29.4% mid / 11.9% tick-floor(>=900us); 17.9% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~3606 distinct values; inversion ratio 0.54
REPORT kstrand
VERDICT: 1 strands across 1 per-CPU kthreads; worst HELD strand 17.0ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#cfa272d9 15 1 17.0 17.0 1 0 tid=0 0%
WORKLOAD cachyos-scx_rusty-primes
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
hot-cpu 15 l2_miss_share 21.8303 HIGH
kthread-strand comm#0dbd544b max_strand_ms 20.9973 MED
long-range-dependent dispatch-rate hurst 0.598899 MED
migration-avalanche migration-rate runs 10 MED
discord 575ms profile_vs_mean 1.58294 MED
kthread-strand comm#dca24885 max_strand_ms 20.0036 MED
kthread-strand comm#0dbd57b1 max_strand_ms 19.003 MED
kthread-strand comm#e3146c3c max_strand_ms 18.9973 MED
kthread-strand comm#ca8154b4 max_strand_ms 17.9981 MED
kthread-strand comm#0357d927 max_strand_ms 17.9961 MED
kthread-strand comm#cfa272d9 max_strand_ms 17.9957 MED
kthread-strand comm#e3146723 max_strand_ms 17.0006 MED
kthread-strand comm#493ffe92 max_strand_ms 16.9999 MED
kthread-strand comm#da1fa6ff max_strand_ms 16.9971 MED
kthread-strand comm#c43b8240 max_strand_ms 15.9995 MED
kthread-strand comm#12f62659 max_strand_ms 14.0009 MED
kthread-strand comm#658ebbd8 max_strand_ms 14.0008 MED
kthread-strand comm#ff850b8f max_strand_ms 12.9843 MED
kthread-strand comm#cfa26f73 max_strand_ms 11.2805 MED
kthread-strand comm#cfa27126 max_strand_ms 10.2962 MED
kthread-strand comm#1beaf0af max_strand_ms 8.00793 MED
kthread-strand comm#dca2436c max_strand_ms 6.00368 MED
long-range-dependent migration-rate hurst 0.581524 MED
CAPTURE complete -- 9333 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 4.3k wake2run; p50 9us p99 3010us p999 11091us worst 41462us; 73.6% fast(<100us) / 12.8% mid / 13.6% tick-floor(>=900us); 2.5% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~4224 distinct values; inversion ratio 0.64
REPORT kstrand
VERDICT: 47 strands across 18 per-CPU kthreads; worst HELD strand 21.0ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#0dbd544b 7 1 21.0 21.0 1 0 tid=0 0%
comm#dca24885 9 7 20.0 20.0 7 0 tid=0 0%
comm#0dbd57b1 7 3 19.0 19.0 3 0 tid=0 0%
comm#e3146c3c 11 1 19.0 19.0 1 0 tid=0 0%
comm#ca8154b4 14 4 18.0 18.0 4 0 tid=0 0%
comm#0357d927 1 1 18.0 18.0 1 0 tid=0 0%
comm#cfa272d9 15 11 18.0 18.0 11 0 tid=0 0%
comm#e3146723 11 2 17.0 17.0 2 0 tid=0 0%
comm#493ffe92 10 1 17.0 17.0 1 0 tid=0 0%
comm#da1fa6ff 13 3 17.0 17.0 3 0 tid=0 0%
comm#c43b8240 12 2 16.0 16.0 2 0 tid=0 0%
comm#12f62659 6 2 14.0 14.0 2 0 tid=0 0%
comm#658ebbd8 0 4 14.0 14.0 4 0 tid=0 0%
comm#ff850b8f 2 1 13.0 13.0 1 0 tid=0 0%
comm#cfa26f73 15 1 11.3 11.3 1 0 tid=0 0%
comm#cfa27126 15 1 10.3 10.3 1 0 tid=0 0%
comm#1beaf0af 4 1 8.0 8.0 1 0 tid=0 0%
comm#dca2436c 9 1 6.0 6.0 1 0 tid=0 0%
WORKLOAD cachyos-scx_rusty-stress-ng-cpu-cache-mem
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
kthread-strand comm#cfa272d9 max_strand_ms 24.2573 MED
long-range-dependent migration-rate hurst 0.863495 MED
migration-avalanche migration-rate runs 79 MED
discord 10292ms profile_vs_mean 1.72487 MED
kthread-strand comm#ff850b8f max_strand_ms 20.983 MED
kthread-strand comm#dca24885 max_strand_ms 20.9678 MED
kthread-strand comm#493ffe92 max_strand_ms 20.9665 MED
kthread-strand comm#da1fa6ff max_strand_ms 20.0044 MED
kthread-strand comm#fa4c2d9c max_strand_ms 20.0005 MED
kthread-strand comm#c43b8240 max_strand_ms 20.0003 MED
kthread-strand comm#0dbd544b max_strand_ms 20.0002 MED
kthread-strand comm#658ebbd8 max_strand_ms 20.0001 MED
kthread-strand comm#e1c397f1 max_strand_ms 20.0001 MED
kthread-strand comm#12f62659 max_strand_ms 20 MED
kthread-strand comm#0357d927 max_strand_ms 19.9998 MED
kthread-strand comm#ca8154b4 max_strand_ms 19.999 MED
kthread-strand comm#180cd039 max_strand_ms 19.975 MED
kthread-strand comm#2f099b5a max_strand_ms 18.9991 MED
kthread-strand comm#19c36834 max_strand_ms 16.9996 MED
kthread-strand comm#e3146723 max_strand_ms 16.9994 MED
kthread-strand comm#a3cc20ce max_strand_ms 16.9993 MED
kthread-strand comm#fa4c2883 max_strand_ms 16.9864 MED
kthread-strand comm#70339bdc max_strand_ms 16.1247 MED
kthread-strand comm#1beaf0af max_strand_ms 15.9991 MED
kthread-strand comm#58be85c4 max_strand_ms 15.9982 MED
kthread-strand comm#65897344 max_strand_ms 14.9983 MED
kthread-strand comm#cfa26f73 max_strand_ms 12.1589 MED
kthread-strand comm#494058a1 max_strand_ms 6.99823 MED
long-range-dependent dispatch-rate hurst 0.834562 MED
hot-cpu 3 l2_miss_share 6.70406 LOW
CAPTURE complete -- 57302 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 44.0k wake2run; p50 5us p99 2014us p999 13009us worst 96812us; 87.5% fast(<100us) / 2.2% mid / 10.3% tick-floor(>=900us); 5.1% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~43964 distinct values; inversion ratio 0.41
REPORT kstrand
VERDICT: 210 strands across 25 per-CPU kthreads; worst HELD strand 24.3ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#cfa272d9 15 19 24.3 24.3 19 0 tid=0 0%
comm#ff850b8f 2 8 21.0 21.0 8 0 tid=0 0%
comm#dca24885 9 13 21.0 21.0 13 0 tid=0 0%
comm#493ffe92 10 8 21.0 21.0 8 0 tid=0 0%
comm#da1fa6ff 13 8 20.0 20.0 8 0 tid=0 0%
comm#fa4c2d9c 3 13 20.0 20.0 13 0 tid=0 0%
comm#c43b8240 12 25 20.0 20.0 25 0 tid=0 0%
comm#0dbd544b 7 31 20.0 20.0 31 0 tid=0 0%
comm#658ebbd8 0 26 20.0 20.0 26 0 tid=0 0%
comm#e1c397f1 8 4 20.0 20.0 4 0 tid=0 0%
comm#12f62659 6 5 20.0 20.0 5 0 tid=0 0%
comm#0357d927 1 5 20.0 20.0 5 0 tid=0 0%
comm#ca8154b4 14 10 20.0 20.0 10 0 tid=0 0%
comm#180cd039 5 13 20.0 20.0 13 0 tid=0 0%
comm#2f099b5a 2 1 19.0 19.0 1 0 tid=0 0%
comm#19c36834 14 1 17.0 17.0 1 0 tid=0 0%
comm#e3146723 11 8 17.0 17.0 8 0 tid=0 0%
comm#a3cc20ce 13 1 17.0 17.0 1 0 tid=0 0%
comm#fa4c2883 3 1 17.0 17.0 1 0 tid=0 0%
comm#70339bdc 4 1 16.1 16.1 1 0 tid=0 0%
WORKLOAD cachyos-scx_rusty-xz-compression
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
kthread-strand comm#cfa26f73 max_strand_ms 39.9962 MED
hot-waker 132269 wakes_issued 31 MED
long-range-dependent dispatch-rate hurst 0.658497 MED
migration-avalanche migration-rate runs 15 MED
discord 1081ms profile_vs_mean 1.65527 MED
kthread-strand comm#fa4c2d9c max_strand_ms 20.0038 MED
kthread-strand comm#c43b8240 max_strand_ms 20.0001 MED
kthread-strand comm#e1c397f1 max_strand_ms 19.9995 MED
kthread-strand comm#1beaf0af max_strand_ms 19.999 MED
kthread-strand comm#da1fa6ff max_strand_ms 19.999 MED
kthread-strand comm#180cd039 max_strand_ms 19.9981 MED
kthread-strand comm#dca24885 max_strand_ms 19.9975 MED
kthread-strand comm#ff850b8f max_strand_ms 19.9962 MED
kthread-strand comm#658ebbd8 max_strand_ms 18.9981 MED
kthread-strand comm#cfa272d9 max_strand_ms 18.9971 MED
kthread-strand comm#e3146723 max_strand_ms 18.9303 MED
kthread-strand comm#0dbd544b max_strand_ms 18.0016 MED
kthread-strand comm#12f62659 max_strand_ms 17.9992 MED
kthread-strand comm#493ffe92 max_strand_ms 16.9984 MED
kthread-strand comm#ca8154b4 max_strand_ms 12.9995 MED
kthread-strand comm#0357d927 max_strand_ms 11.9942 MED
long-range-dependent migration-rate hurst 0.59368 MED
ioctl-slow 132269/xz duration_ms 0.00074 LOW
hot-cpu 10 l2_miss_share 7.64118 LOW
ioctl-slow 132269/xz duration_ms 0.00055 LOW
CAPTURE complete -- 65522 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 3.2k wake2run; p50 24us p99 8314us p999 15162us worst 20030us; 68.8% fast(<100us) / 12.8% mid / 18.3% tick-floor(>=900us); 7.5% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~3232 distinct values; inversion ratio 0.57
REPORT kstrand
VERDICT: 111 strands across 17 per-CPU kthreads; worst HELD strand 40.0ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#cfa26f73 15 25 40.0 40.0 25 0 tid=0 0%
comm#fa4c2d9c 3 11 20.0 20.0 11 0 tid=0 0%
comm#c43b8240 12 10 20.0 20.0 10 0 tid=0 0%
comm#e1c397f1 8 4 20.0 20.0 4 0 tid=0 0%
comm#1beaf0af 4 2 20.0 20.0 2 0 tid=0 0%
comm#da1fa6ff 13 5 20.0 20.0 5 0 tid=0 0%
comm#180cd039 5 2 20.0 20.0 2 0 tid=0 0%
comm#dca24885 9 7 20.0 20.0 7 0 tid=0 0%
comm#ff850b8f 2 3 20.0 20.0 3 0 tid=0 0%
comm#658ebbd8 0 11 19.0 19.0 11 0 tid=0 0%
comm#cfa272d9 15 2 19.0 19.0 2 0 tid=0 0%
comm#e3146723 11 4 18.9 18.9 4 0 tid=0 0%
comm#0dbd544b 7 13 18.0 18.0 13 0 tid=0 0%
comm#12f62659 6 2 18.0 18.0 2 0 tid=0 0%
comm#493ffe92 10 5 17.0 17.0 5 0 tid=0 0%
comm#ca8154b4 14 4 13.0 13.0 4 0 tid=0 0%
comm#0357d927 1 1 12.0 12.0 1 0 tid=0 0%
WORKLOAD cachyos-scx_tickless-primes
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
discord 183ms profile_vs_mean 3.90416 HIGH
hot-cpu 15 l2_miss_share 28.5203 HIGH
kthread-strand comm#e1c397f1 max_strand_ms 21.0307 MED
migration-avalanche migration-rate runs 60 MED
kthread-strand comm#658ebbd8 max_strand_ms 21.0032 MED
kthread-strand comm#0357d927 max_strand_ms 20.9718 MED
kthread-strand comm#0dbd57b1 max_strand_ms 20.9689 MED
kthread-strand comm#dca24885 max_strand_ms 20.9669 MED
kthread-strand comm#493ffe92 max_strand_ms 20.9668 MED
kthread-strand comm#fa4c2883 max_strand_ms 16.9999 MED
kthread-strand comm#e3146723 max_strand_ms 15.9615 MED
kthread-strand comm#ff850b8f max_strand_ms 13.0001 MED
kthread-strand comm#cfa26f73 max_strand_ms 12.3797 MED
kthread-strand comm#cfa272d9 max_strand_ms 10.0025 MED
kthread-strand comm#ca8154b4 max_strand_ms 5.0796 MED
kthread-strand comm#1beaf0af max_strand_ms 5.01411 MED
CAPTURE complete -- 6132 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 1.1k wake2run; p50 14us p99 20010us p999 21006us worst 21010us; 81.3% fast(<100us) / 7.5% mid / 11.2% tick-floor(>=900us); 35.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~1147 distinct values; inversion ratio 0.57
REPORT kstrand
VERDICT: 26 strands across 13 per-CPU kthreads; worst HELD strand 21.0ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#e1c397f1 8 4 21.0 21.0 4 0 tid=0 0%
comm#658ebbd8 0 7 21.0 21.0 7 0 tid=0 0%
comm#0357d927 1 1 21.0 21.0 1 0 tid=0 0%
comm#0dbd57b1 7 1 21.0 21.0 1 0 tid=0 0%
comm#dca24885 9 1 21.0 21.0 1 0 tid=0 0%
comm#493ffe92 10 1 21.0 21.0 1 0 tid=0 0%
comm#fa4c2883 3 4 17.0 17.0 4 0 tid=0 0%
comm#e3146723 11 1 16.0 16.0 1 0 tid=0 0%
comm#ff850b8f 2 1 13.0 13.0 1 0 tid=0 0%
comm#cfa26f73 15 1 12.4 12.4 1 0 tid=0 0%
comm#cfa272d9 15 2 10.0 10.0 2 0 tid=0 0%
comm#ca8154b4 14 1 5.1 5.1 1 0 tid=0 0%
comm#1beaf0af 4 1 5.0 5.0 1 0 tid=0 0%
WORKLOAD cachyos-scx_tickless-stress-ng-cpu-cache-mem
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
kthread-strand comm#658ebbd8 max_strand_ms 21.9992 MED
long-range-dependent dispatch-rate hurst 0.880664 MED
migration-avalanche migration-rate runs 644 MED
discord 9478ms profile_vs_mean 1.51466 MED
kthread-strand comm#cfa26f73 max_strand_ms 21.5105 MED
kthread-strand comm#ca8154b4 max_strand_ms 21.2589 MED
kthread-strand comm#e1c397f1 max_strand_ms 21.0109 MED
kthread-strand comm#493ffe92 max_strand_ms 21.0015 MED
kthread-strand comm#fa4c2883 max_strand_ms 21.0001 MED
kthread-strand comm#d1030141 max_strand_ms 20.9977 MED
kthread-strand comm#da1fa6ff max_strand_ms 20.9717 MED
kthread-strand comm#e3146723 max_strand_ms 20.9694 MED
kthread-strand comm#1beaf0af max_strand_ms 20.9552 MED
kthread-strand comm#180cd039 max_strand_ms 20.9512 MED
kthread-strand comm#0dbd57b1 max_strand_ms 20.95 MED
kthread-strand comm#c43b8240 max_strand_ms 20.9401 MED
kthread-strand comm#0357d927 max_strand_ms 20.9241 MED
kthread-strand comm#ff850b8f max_strand_ms 20.9167 MED
kthread-strand comm#dca24885 max_strand_ms 20.8991 MED
kthread-strand comm#cfa272d9 max_strand_ms 13.51 MED
kthread-strand comm#2f099b5a max_strand_ms 9.00771 MED
kthread-strand comm#12f62659 max_strand_ms 7.99334 MED
long-range-dependent migration-rate hurst 0.872065 MED
hot-cpu 3 l2_miss_share 6.79893 LOW
CAPTURE complete -- 34204 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 19.4k wake2run; p50 8us p99 11042us p999 20095us worst 21110us; 88.5% fast(<100us) / 4.6% mid / 6.9% tick-floor(>=900us); 39.9% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~19094 distinct values; inversion ratio 0.36
REPORT kstrand
VERDICT: 202 strands across 19 per-CPU kthreads; worst HELD strand 22.0ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#658ebbd8 0 44 22.0 22.0 44 0 tid=0 0%
comm#cfa26f73 15 13 21.5 21.5 13 0 tid=0 0%
comm#ca8154b4 14 7 21.3 21.3 7 0 tid=0 0%
comm#e1c397f1 8 15 21.0 21.0 15 0 tid=0 0%
comm#493ffe92 10 7 21.0 21.0 7 0 tid=0 0%
comm#fa4c2883 3 28 21.0 21.0 28 0 tid=0 0%
comm#d1030141 15 1 21.0 21.0 1 0 tid=0 0%
comm#da1fa6ff 13 5 21.0 21.0 5 0 tid=0 0%
comm#e3146723 11 7 21.0 21.0 7 0 tid=0 0%
comm#1beaf0af 4 14 21.0 21.0 14 0 tid=0 0%
comm#180cd039 5 5 21.0 21.0 5 0 tid=0 0%
comm#0dbd57b1 7 16 20.9 20.9 16 0 tid=0 0%
comm#c43b8240 12 7 20.9 20.9 7 0 tid=0 0%
comm#0357d927 1 3 20.9 20.9 3 0 tid=0 0%
comm#ff850b8f 2 20 20.9 20.9 20 0 tid=0 0%
comm#dca24885 9 5 20.9 20.9 5 0 tid=0 0%
comm#cfa272d9 15 2 13.5 13.5 2 0 tid=0 0%
comm#2f099b5a 2 2 9.0 9.0 2 0 tid=0 0%
comm#12f62659 6 1 8.0 8.0 1 0 tid=0 0%
WORKLOAD cachyos-scx_tickless-xz-compression
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
kthread-strand comm#fa4c2883 max_strand_ms 21.9963 MED
migration-avalanche migration-rate runs 109 MED
discord 496ms profile_vs_mean 1.52705 MED
kthread-strand comm#e1c397f1 max_strand_ms 21.0075 MED
kthread-strand comm#1beaf0af max_strand_ms 20.9995 MED
kthread-strand comm#658ebbd8 max_strand_ms 20.9995 MED
kthread-strand comm#ff850b8f max_strand_ms 20.9993 MED
kthread-strand comm#ca8154b4 max_strand_ms 20.9992 MED
kthread-strand comm#c43b8240 max_strand_ms 20.9992 MED
kthread-strand comm#493ffe92 max_strand_ms 20.9983 MED
kthread-strand comm#e3146723 max_strand_ms 20.9979 MED
kthread-strand comm#180cd039 max_strand_ms 20.995 MED
kthread-strand comm#0dbd57b1 max_strand_ms 20.9946 MED
kthread-strand comm#12f62659 max_strand_ms 20.994 MED
kthread-strand comm#da1fa6ff max_strand_ms 20.9917 MED
kthread-strand comm#ae46779d max_strand_ms 19.1625 MED
kthread-strand comm#cfa272d9 max_strand_ms 18.4367 MED
kthread-strand comm#658123d8 max_strand_ms 16.9985 MED
kthread-strand comm#0357d927 max_strand_ms 14.734 MED
kthread-strand comm#cfa26f73 max_strand_ms 14.0483 MED
kthread-strand comm#dca24885 max_strand_ms 6.995 MED
ioctl-slow 133421/xz duration_ms 0.00054 LOW
hot-cpu 1 l2_miss_share 6.86221 LOW
ioctl-slow 133421/xz duration_ms 0.00039 LOW
CAPTURE complete -- 64937 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 2.6k wake2run; p50 37us p99 10082us p999 20985us worst 21053us; 64.9% fast(<100us) / 20.3% mid / 14.8% tick-floor(>=900us); 26.6% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~2622 distinct values; inversion ratio 0.56
REPORT kstrand
VERDICT: 105 strands across 19 per-CPU kthreads; worst HELD strand 22.0ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#fa4c2883 3 14 22.0 22.0 14 0 tid=0 0%
comm#e1c397f1 8 8 21.0 21.0 8 0 tid=0 0%
comm#1beaf0af 4 10 21.0 21.0 10 0 tid=0 0%
comm#658ebbd8 0 23 21.0 21.0 23 0 tid=0 0%
comm#ff850b8f 2 5 21.0 21.0 5 0 tid=0 0%
comm#ca8154b4 14 4 21.0 21.0 4 0 tid=0 0%
comm#c43b8240 12 4 21.0 21.0 4 0 tid=0 0%
comm#493ffe92 10 2 21.0 21.0 2 0 tid=0 0%
comm#e3146723 11 2 21.0 21.0 2 0 tid=0 0%
comm#180cd039 5 4 21.0 21.0 4 0 tid=0 0%
comm#0dbd57b1 7 7 21.0 21.0 7 0 tid=0 0%
comm#12f62659 6 4 21.0 21.0 4 0 tid=0 0%
comm#da1fa6ff 13 2 21.0 21.0 2 0 tid=0 0%
comm#ae46779d 1 2 19.2 19.2 2 0 tid=0 0%
comm#cfa272d9 15 7 18.4 18.4 7 0 tid=0 0%
comm#658123d8 0 1 17.0 17.0 1 0 tid=0 0%
comm#0357d927 1 3 14.7 14.7 3 0 tid=0 0%
comm#cfa26f73 15 2 14.0 14.0 2 0 tid=0 0%
comm#dca24885 9 1 7.0 7.0 1 0 tid=0 0%
WORKLOAD cold-wake-16c-EEVDF
THERMAL/POWER
cpu temp peak 53.2 C avg 51.7 C
power avg 79.5 W peak 106.8 W
idle C2 avg 94.3% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
hot-cpu 1 l2_miss_share 82.6974 HIGH
long-range-dependent dispatch-rate hurst 0.531044 MED
CAPTURE complete -- 848 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 61 wake2run; p50 3us p99 7us p999 7us worst 7us; 100.0% fast(<100us) / 0.0% mid / 0.0% tick-floor(>=900us); 0.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~52 distinct values; inversion ratio 0.52
COLD-WAKE: cold-cache worst 4.6x at 32KB (0.03ms cold); CPU freq ramp worst +38%
WORKLOAD cold-wake-16c-PANDEMONIUM
THERMAL/POWER
cpu temp peak 51.5 C avg 51.1 C
power avg 81.0 W peak 102.4 W
idle C2 avg 94.1% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
kthread-strand comm#12f62659 max_strand_ms 505.683 HIGH
hot-cpu 1 l2_miss_share 81.2772 HIGH
long-range-dependent dispatch-rate hurst 0.604337 MED
CAPTURE complete -- 888 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 111 wake2run; p50 6us p99 127us p999 133us worst 133us; 98.2% fast(<100us) / 1.8% mid / 0.0% tick-floor(>=900us); 0.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~111 distinct values; inversion ratio 0.61
REPORT kstrand
VERDICT: 1 strands across 1 per-CPU kthreads; worst HELD strand 505.7ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#12f62659 6 1 505.7 505.7 1 0 tid=0 0%
COLD-WAKE: cold-cache worst 6.2x at 32KB (0.04ms cold); CPU freq ramp worst +40%
WORKLOAD cold-wake-16c-scx_beerland
THERMAL/POWER
cpu temp peak 54.8 C avg 51.7 C
power avg 79.1 W peak 103.0 W
idle C2 avg 94.3% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
hot-cpu 1 l2_miss_share 82.7545 HIGH
long-range-dependent dispatch-rate hurst 0.544159 MED
CAPTURE complete -- 820 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 62 wake2run; p50 4us p99 1532us p999 1532us worst 1532us; 93.5% fast(<100us) / 3.2% mid / 3.2% tick-floor(>=900us); 0.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~57 distinct values; inversion ratio 0.50
COLD-WAKE: cold-cache worst 4.0x at 32KB (0.03ms cold); CPU freq ramp worst +40%
WORKLOAD cold-wake-16c-scx_bpfland
THERMAL/POWER
cpu temp peak 53.0 C avg 51.3 C
power avg 79.5 W peak 99.7 W
idle C2 avg 94.1% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
hot-cpu 1 l2_miss_share 44.9366 HIGH
long-range-dependent dispatch-rate hurst 0.542044 MED
CAPTURE complete -- 851 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 63 wake2run; p50 3us p99 1927us p999 1927us worst 1927us; 96.8% fast(<100us) / 0.0% mid / 3.2% tick-floor(>=900us); 0.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~51 distinct values; inversion ratio 0.49
COLD-WAKE: cold-cache worst 4.2x at 32KB (0.03ms cold); CPU freq ramp worst +40%
WORKLOAD cold-wake-16c-scx_cake
THERMAL/POWER
cpu temp peak 53.0 C avg 50.6 C
power avg 78.8 W peak 99.5 W
idle C2 avg 94.2% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
hot-cpu 1 l2_miss_share 82.5464 HIGH
kthread-strand comm#0878f233 max_strand_ms 14.1487 MED
long-range-dependent dispatch-rate hurst 0.533535 MED
kthread-strand comm#cfa26f73 max_strand_ms 9.00036 MED
CAPTURE complete -- 843 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 59 wake2run; p50 4us p99 10us p999 10us worst 10us; 100.0% fast(<100us) / 0.0% mid / 0.0% tick-floor(>=900us); 0.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~53 distinct values; inversion ratio 0.52
REPORT kstrand
VERDICT: 3 strands across 2 per-CPU kthreads; worst HELD strand 14.1ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#0878f233 0 2 14.1 14.1 2 0 tid=0 0%
comm#cfa26f73 15 1 9.0 9.0 1 0 tid=0 0%
COLD-WAKE: cold-cache worst 4.1x at 32KB (0.03ms cold); CPU freq ramp worst +40%
WORKLOAD cold-wake-16c-scx_cosmos
THERMAL/POWER
cpu temp peak 55.2 C avg 50.7 C
power avg 78.1 W peak 100.7 W
idle C2 avg 94.4% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
hot-cpu 1 l2_miss_share 82.8817 HIGH
long-range-dependent dispatch-rate hurst 0.546111 MED
CAPTURE complete -- 844 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 64 wake2run; p50 3us p99 1459us p999 1459us worst 1459us; 96.9% fast(<100us) / 1.6% mid / 1.6% tick-floor(>=900us); 0.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~58 distinct values; inversion ratio 0.52
COLD-WAKE: cold-cache worst 4.5x at 32KB (0.03ms cold); CPU freq ramp worst +40%
WORKLOAD cold-wake-16c-scx_flash
THERMAL/POWER
cpu temp peak 54.0 C avg 51.3 C
power avg 79.8 W peak 105.0 W
idle C2 avg 94.4% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
kthread-strand comm#cfa27126 max_strand_ms 138.325 HIGH
hot-cpu 1 l2_miss_share 84.4852 HIGH
mean-reverting dispatch-rate hurst 0.456844 MED
kthread-strand comm#288f3ed3 max_strand_ms 63.2107 MED
kthread-strand comm#ff850676 max_strand_ms 45.59 MED
kthread-strand comm#0357d927 max_strand_ms 42.9247 MED
kthread-strand comm#12f62659 max_strand_ms 26.2698 MED
kthread-strand comm#2f099b5a max_strand_ms 13.6441 MED
kthread-strand comm#0dbd57b1 max_strand_ms 11.8166 MED
kthread-strand comm#180cd039 max_strand_ms 7.23054 MED
CAPTURE complete -- 829 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 33 wake2run; p50 3us p99 279us p999 279us worst 279us; 97.0% fast(<100us) / 3.0% mid / 0.0% tick-floor(>=900us); 0.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~30 distinct values; inversion ratio 0.47
REPORT kstrand
VERDICT: 28 strands across 8 per-CPU kthreads; worst HELD strand 138.3ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#cfa27126 15 14 138.3 138.3 14 0 tid=0 0%
comm#288f3ed3 2 1 63.2 63.2 1 0 tid=0 0%
comm#ff850676 2 5 45.6 45.6 5 0 tid=0 0%
comm#0357d927 1 2 42.9 42.9 2 0 tid=0 0%
comm#12f62659 6 2 26.3 26.3 2 0 tid=0 0%
comm#2f099b5a 2 1 13.6 13.6 1 0 tid=0 0%
comm#0dbd57b1 7 2 11.8 11.8 2 0 tid=0 0%
comm#180cd039 5 1 7.2 7.2 1 0 tid=0 0%
COLD-WAKE: cold-cache worst 4.2x at 32KB (0.03ms cold); CPU freq ramp worst +38%
WORKLOAD cold-wake-16c-scx_flow
THERMAL/POWER
cpu temp peak 55.0 C avg 52.4 C
power avg 81.3 W peak 104.0 W
idle C2 avg 94.2% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
kthread-strand comm#0878f233 max_strand_ms 250.676 HIGH
hot-cpu 1 l2_miss_share 84.0519 HIGH
long-range-dependent dispatch-rate hurst 0.565119 MED
CAPTURE complete -- 844 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 69 wake2run; p50 3us p99 12us p999 12us worst 12us; 100.0% fast(<100us) / 0.0% mid / 0.0% tick-floor(>=900us); 0.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~57 distinct values; inversion ratio 0.43
REPORT kstrand
VERDICT: 1 strands across 1 per-CPU kthreads; worst HELD strand 250.7ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#0878f233 0 1 250.7 250.7 1 0 tid=0 0%
COLD-WAKE: cold-cache worst 3.9x at 32KB (0.03ms cold); CPU freq ramp worst +38%
WORKLOAD cold-wake-16c-scx_lavd
THERMAL/POWER
cpu temp peak 55.0 C avg 51.7 C
power avg 78.4 W peak 106.0 W
idle C2 avg 94.2% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
hot-cpu 1 l2_miss_share 81.1309 HIGH
kthread-strand comm#cfa272d9 max_strand_ms 8.7381 MED
long-range-dependent dispatch-rate hurst 0.546789 MED
kthread-strand comm#fa4c32b5 max_strand_ms 8.54726 MED
kthread-strand comm#0878f080 max_strand_ms 8.01004 MED
kthread-strand comm#fa4c2d9c max_strand_ms 7.5534 MED
kthread-strand comm#cfa26f73 max_strand_ms 7.0197 MED
kthread-strand comm#180cd039 max_strand_ms 7.00188 MED
kthread-strand comm#cfa26dc0 max_strand_ms 6.00181 MED
CAPTURE complete -- 864 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 60 wake2run; p50 4us p99 10us p999 10us worst 10us; 100.0% fast(<100us) / 0.0% mid / 0.0% tick-floor(>=900us); 0.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~56 distinct values; inversion ratio 0.50
REPORT kstrand
VERDICT: 19 strands across 7 per-CPU kthreads; worst HELD strand 8.7ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#cfa272d9 15 1 8.7 8.7 1 0 tid=0 0%
comm#fa4c32b5 3 8 8.5 8.5 8 0 tid=0 0%
comm#0878f080 0 1 8.0 8.0 1 0 tid=0 0%
comm#fa4c2d9c 3 1 7.6 7.6 1 0 tid=0 0%
comm#cfa26f73 15 6 7.0 7.0 6 0 tid=0 0%
comm#180cd039 5 1 7.0 7.0 1 0 tid=0 0%
comm#cfa26dc0 15 1 6.0 6.0 1 0 tid=0 0%
COLD-WAKE: cold-cache worst 4.4x at 32KB (0.03ms cold); CPU freq ramp worst +40%
WORKLOAD cold-wake-16c-scx_p2dq
THERMAL/POWER
cpu temp peak 54.8 C avg 51.2 C
power avg 81.3 W peak 114.4 W
idle C2 avg 94.2% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
hot-cpu 1 l2_miss_share 81.9766 HIGH
kthread-strand comm#0878f233 max_strand_ms 20.0006 MED
long-range-dependent dispatch-rate hurst 0.527386 MED
kthread-strand comm#cfa27126 max_strand_ms 19.996 MED
kthread-strand comm#0357d927 max_strand_ms 19.0004 MED
kthread-strand comm#ff850676 max_strand_ms 18.9847 MED
kthread-strand comm#dca24885 max_strand_ms 17.9869 MED
kthread-strand comm#fa4c2d9c max_strand_ms 15.9903 MED
kthread-strand comm#e3146c3c max_strand_ms 12.517 MED
kthread-strand comm#e1c397f1 max_strand_ms 11.0001 MED
kthread-strand comm#d4e6d4f1 max_strand_ms 9.0014 MED
kthread-strand comm#493ffe92 max_strand_ms 7.30014 MED
CAPTURE complete -- 884 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 55 wake2run; p50 4us p99 4345us p999 4345us worst 4345us; 98.2% fast(<100us) / 0.0% mid / 1.8% tick-floor(>=900us); 0.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~50 distinct values; inversion ratio 0.38
REPORT kstrand
VERDICT: 58 strands across 10 per-CPU kthreads; worst HELD strand 20.0ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#0878f233 0 23 20.0 20.0 23 0 tid=0 0%
comm#cfa27126 15 10 20.0 20.0 10 0 tid=0 0%
comm#0357d927 1 5 19.0 19.0 5 0 tid=0 0%
comm#ff850676 2 3 19.0 19.0 3 0 tid=0 0%
comm#dca24885 9 2 18.0 18.0 2 0 tid=0 0%
comm#fa4c2d9c 3 7 16.0 16.0 7 0 tid=0 0%
comm#e3146c3c 11 1 12.5 12.5 1 0 tid=0 0%
comm#e1c397f1 8 5 11.0 11.0 5 0 tid=0 0%
comm#d4e6d4f1 12 1 9.0 9.0 1 0 tid=0 0%
comm#493ffe92 10 1 7.3 7.3 1 0 tid=0 0%
COLD-WAKE: cold-cache worst 5.7x at 32KB (0.04ms cold); CPU freq ramp worst +40%
WORKLOAD cold-wake-16c-scx_rustland
THERMAL/POWER
cpu temp peak 54.0 C avg 51.2 C
power avg 78.8 W peak 107.8 W
idle C2 avg 94.2% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
hot-cpu 3 l2_miss_share 32.0255 HIGH
kthread-strand comm#cfa26f73 max_strand_ms 13 MED
long-range-dependent dispatch-rate hurst 0.555596 MED
discord 31644ms profile_vs_mean 1.61639 MED
CAPTURE complete -- 3600 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 73 wake2run; p50 10us p99 1294us p999 1294us worst 1294us; 97.3% fast(<100us) / 1.4% mid / 1.4% tick-floor(>=900us); 0.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~73 distinct values; inversion ratio 0.50
REPORT kstrand
VERDICT: 2 strands across 1 per-CPU kthreads; worst HELD strand 13.0ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#cfa26f73 15 2 13.0 13.0 2 0 tid=0 0%
COLD-WAKE: cold-cache worst 4.3x at 32KB (0.03ms cold); CPU freq ramp worst +38%
WORKLOAD cold-wake-16c-scx_rusty
THERMAL/POWER
cpu temp peak 53.0 C avg 51.2 C
power avg 80.3 W peak 111.5 W
idle C2 avg 94.2% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
hot-cpu 1 l2_miss_share 83.1471 HIGH
kthread-strand comm#0357d927 max_strand_ms 20.0002 MED
long-range-dependent dispatch-rate hurst 0.538866 MED
kthread-strand comm#0878f080 max_strand_ms 19.9995 MED
kthread-strand comm#e1c3948b max_strand_ms 19.9981 MED
kthread-strand comm#cfa26f73 max_strand_ms 19.9959 MED
kthread-strand comm#dca24885 max_strand_ms 19.0049 MED
kthread-strand comm#e1c397f1 max_strand_ms 18.9911 MED
kthread-strand comm#fa4c32b5 max_strand_ms 18.0063 MED
kthread-strand comm#ff850676 max_strand_ms 18.0044 MED
kthread-strand comm#cfa27126 max_strand_ms 15.2801 MED
kthread-strand comm#cfa26dc0 max_strand_ms 12.2917 MED
kthread-strand comm#e3146c3c max_strand_ms 12.0031 MED
kthread-strand comm#d4e6d4f1 max_strand_ms 7.99993 MED
kthread-strand comm#12f62659 max_strand_ms 7.00081 MED
kthread-strand comm#2f099b5a max_strand_ms 5.999 MED
CAPTURE complete -- 920 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 58 wake2run; p50 4us p99 66us p999 66us worst 66us; 100.0% fast(<100us) / 0.0% mid / 0.0% tick-floor(>=900us); 0.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~52 distinct values; inversion ratio 0.53
REPORT kstrand
VERDICT: 83 strands across 14 per-CPU kthreads; worst HELD strand 20.0ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#0357d927 1 17 20.0 20.0 17 0 tid=0 0%
comm#0878f080 0 13 20.0 20.0 13 0 tid=0 0%
comm#e1c3948b 8 1 20.0 20.0 1 0 tid=0 0%
comm#cfa26f73 15 17 20.0 20.0 17 0 tid=0 0%
comm#dca24885 9 3 19.0 19.0 3 0 tid=0 0%
comm#e1c397f1 8 10 19.0 19.0 10 0 tid=0 0%
comm#fa4c32b5 3 2 18.0 18.0 2 0 tid=0 0%
comm#ff850676 2 11 18.0 18.0 11 0 tid=0 0%
comm#cfa27126 15 1 15.3 15.3 1 0 tid=0 0%
comm#cfa26dc0 15 1 12.3 12.3 1 0 tid=0 0%
comm#e3146c3c 11 2 12.0 12.0 2 0 tid=0 0%
comm#d4e6d4f1 12 1 8.0 8.0 1 0 tid=0 0%
comm#12f62659 6 3 7.0 7.0 3 0 tid=0 0%
comm#2f099b5a 2 1 6.0 6.0 1 0 tid=0 0%
COLD-WAKE: cold-cache worst 4.6x at 32KB (0.03ms cold); CPU freq ramp worst +38%
WORKLOAD cold-wake-16c-scx_tickless
THERMAL/POWER
cpu temp peak 53.0 C avg 51.0 C
power avg 79.2 W peak 102.7 W
idle C2 avg 92.7% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
hot-cpu 1 l2_miss_share 79.6173 HIGH
kthread-strand comm#ff850676 max_strand_ms 21.0053 MED
kthread-strand comm#0878f233 max_strand_ms 21.0028 MED
kthread-strand comm#2f099b5a max_strand_ms 21.0002 MED
kthread-strand comm#cfa26f73 max_strand_ms 20.9999 MED
kthread-strand comm#fa4c2d9c max_strand_ms 20.9995 MED
kthread-strand comm#1beaeefc max_strand_ms 19.0038 MED
kthread-strand comm#cfa27126 max_strand_ms 8.53231 MED
CAPTURE complete -- 936 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 54 wake2run; p50 3us p99 25us p999 25us worst 25us; 100.0% fast(<100us) / 0.0% mid / 0.0% tick-floor(>=900us); 0.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~44 distinct values; inversion ratio 0.52
REPORT kstrand
VERDICT: 103 strands across 7 per-CPU kthreads; worst HELD strand 21.0ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#ff850676 2 30 21.0 21.0 30 0 tid=0 0%
comm#0878f233 0 45 21.0 21.0 45 0 tid=0 0%
comm#2f099b5a 2 2 21.0 21.0 2 0 tid=0 0%
comm#cfa26f73 15 20 21.0 21.0 20 0 tid=0 0%
comm#fa4c2d9c 3 4 21.0 21.0 4 0 tid=0 0%
comm#1beaeefc 4 1 19.0 19.0 1 0 tid=0 0%
comm#cfa27126 15 1 8.5 8.5 1 0 tid=0 0%
COLD-WAKE: cold-cache worst 4.4x at 32KB (0.03ms cold); CPU freq ramp worst +40%
WORKLOAD cold-wake-starve-16c-EEVDF
THERMAL/POWER
cpu temp peak 59.2 C avg 54.1 C
power avg 89.5 W peak 108.5 W
idle C2 avg 92.4% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
kthread-strand comm#cfa26dc0 max_strand_ms 448.003 HIGH
hot-cpu 3 l2_miss_share 62.1431 HIGH
mid-trace-death comm#e3e5febe/276784 before_end_s 35.0414 MED
long-range-dependent dispatch-rate hurst 0.924162 MED
discord 49054ms profile_vs_mean 2.92707 MED
mid-trace-death comm#e3e5febe/276817 before_end_s 28.0214 MED
mid-trace-death comm#e3e5febe/276899 before_end_s 7.01981 MED
CAPTURE complete -- 212800 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 95.3k wake2run; p50 1us p99 9us p999 49us worst 894us; 100.0% fast(<100us) / 0.0% mid / 0.0% tick-floor(>=900us); 0.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~59570 distinct values; inversion ratio 0.69
LOCATED: 2 structured region(s) FEW_UNIQUE[23%..24%], FEW_UNIQUE[30%..30%]
REPORT kstrand
VERDICT: 1 strands across 1 per-CPU kthreads; worst HELD strand 448.0ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#cfa26dc0 15 1 448.0 448.0 1 0 tid=0 0%
WORKLOAD cold-wake-starve-16c-PANDEMONIUM
THERMAL/POWER
cpu temp peak 60.2 C avg 53.4 C
power avg 90.9 W peak 109.5 W
idle C2 avg 92.7% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
hot-cpu 15 l2_miss_share 35.3773 HIGH
mid-trace-death comm#e3e5febe/277365 before_end_s 35.0258 MED
long-range-dependent dispatch-rate hurst 0.919768 MED
discord 21010ms profile_vs_mean 2.86957 MED
mid-trace-death comm#e3e5febe/277394 before_end_s 28.0057 MED
mid-trace-death comm#e3e5febe/277479 before_end_s 7.00079 MED
CAPTURE complete -- 215022 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 97.6k wake2run; p50 2us p99 4us p999 8us worst 2017us; 100.0% fast(<100us) / 0.0% mid / 0.0% tick-floor(>=900us); 0.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~76268 distinct values; inversion ratio 0.68
LOCATED: 1 structured region(s) FEW_UNIQUE[24%..24%]
WORKLOAD cold-wake-starve-16c-scx_beerland
THERMAL/POWER
cpu temp peak 58.6 C avg 53.7 C
power avg 91.7 W peak 108.4 W
idle C2 avg 92.5% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
discord 43347ms profile_vs_mean 3.01786 HIGH
hot-cpu 15 l2_miss_share 38.1546 HIGH
mid-trace-death comm#e3e5febe/283843 before_end_s 35.0242 MED
long-range-dependent dispatch-rate hurst 0.935343 MED
mid-trace-death comm#e3e5febe/283874 before_end_s 28.0051 MED
mid-trace-death comm#e3e5febe/283959 before_end_s 7.00152 MED
CAPTURE complete -- 211079 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 94.5k wake2run; p50 1us p99 947us p999 947us worst 1949us; 98.9% fast(<100us) / 0.1% mid / 1.1% tick-floor(>=900us); 0.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~62011 distinct values; inversion ratio 0.69
LOCATED: 6 structured region(s) FEW_UNIQUE[21%..22%], FEW_UNIQUE[25%..26%], FEW_UNIQUE[28%..30%], FEW_UNIQUE[31%..32%], FEW_UNIQUE[33%..34%] +1 more
WORKLOAD cold-wake-starve-16c-scx_bpfland
THERMAL/POWER
cpu temp peak 59.4 C avg 54.1 C
power avg 92.1 W peak 110.0 W
idle C2 avg 92.4% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
discord 21894ms profile_vs_mean 3.4396 HIGH
hot-cpu 3 l2_miss_share 61.0982 HIGH
kthread-strand comm#cfa26f73 max_strand_ms 12 MED
mid-trace-death comm#e3e5febe/277955 before_end_s 35.0422 MED
long-range-dependent dispatch-rate hurst 0.912068 MED
mid-trace-death comm#e3e5febe/277985 before_end_s 28.0242 MED
mid-trace-death comm#e3e5febe/278066 before_end_s 7.01612 MED
CAPTURE complete -- 203851 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 89.7k wake2run; p50 2us p99 948us p999 1948us worst 2039us; 95.6% fast(<100us) / 0.1% mid / 4.4% tick-floor(>=900us); 0.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~58892 distinct values; inversion ratio 0.69
LOCATED: 2 structured region(s) FEW_UNIQUE[28%..29%], FEW_UNIQUE[30%..30%]
REPORT kstrand
VERDICT: 2 strands across 1 per-CPU kthreads; worst HELD strand 12.0ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#cfa26f73 15 2 12.0 12.0 2 0 tid=0 0%
WORKLOAD cold-wake-starve-16c-scx_cake
THERMAL/POWER
cpu temp peak 58.9 C avg 53.3 C
power avg 89.6 W peak 112.6 W
idle C2 avg 92.6% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
discord 0ms profile_vs_mean 5.97085 HIGH
hot-cpu 3 l2_miss_share 41.7165 HIGH
mid-trace-death comm#e3e5febe/282687 before_end_s 35.0041 MED
long-range-dependent dispatch-rate hurst 0.920201 MED
mid-trace-death comm#e3e5febe/282718 before_end_s 28.0022 MED
mid-trace-death comm#e3e5febe/282806 before_end_s 7.00051 MED
CAPTURE complete -- 214697 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 97.2k wake2run; p50 2us p99 5us p999 11us worst 1999us; 100.0% fast(<100us) / 0.0% mid / 0.0% tick-floor(>=900us); 0.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~69860 distinct values; inversion ratio 0.66
LOCATED: 1 structured region(s) FEW_UNIQUE[23%..24%]
WORKLOAD cold-wake-starve-16c-scx_cosmos
THERMAL/POWER
cpu temp peak 58.9 C avg 53.3 C
power avg 89.3 W peak 108.7 W
idle C2 avg 93.0% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
discord 24084ms profile_vs_mean 3.56378 HIGH
hot-cpu 15 l2_miss_share 36.0924 HIGH
kthread-strand comm#cfa27126 max_strand_ms 6.9997 MED
mid-trace-death comm#e3e5febe/282103 before_end_s 35.0458 MED
long-range-dependent dispatch-rate hurst 0.91131 MED
mid-trace-death comm#e3e5febe/282130 before_end_s 28.0278 MED
kthread-strand comm#cfa272d9 max_strand_ms 5.99828 MED
mid-trace-death comm#e3e5febe/282220 before_end_s 7.01622 MED
CAPTURE complete -- 203734 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 89.7k wake2run; p50 2us p99 948us p999 1947us worst 2041us; 95.6% fast(<100us) / 0.0% mid / 4.4% tick-floor(>=900us); 0.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~64450 distinct values; inversion ratio 0.67
LOCATED: 1 structured region(s) FEW_UNIQUE[92%..92%]
REPORT kstrand
VERDICT: 2 strands across 2 per-CPU kthreads; worst HELD strand 7.0ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#cfa27126 15 1 7.0 7.0 1 0 tid=0 0%
comm#cfa272d9 15 1 6.0 6.0 1 0 tid=0 0%
WORKLOAD cold-wake-starve-16c-scx_flash
THERMAL/POWER
cpu temp peak 59.0 C avg 54.0 C
power avg 88.8 W peak 112.1 W
idle C2 avg 92.6% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
kthread-strand comm#cfa27126 max_strand_ms 130.433 HIGH
hot-cpu 3 l2_miss_share 70.9278 HIGH
mid-trace-death comm#e3e5febe/283275 before_end_s 35.0329 MED
long-range-dependent dispatch-rate hurst 0.915184 MED
discord 52096ms profile_vs_mean 2.72988 MED
kthread-strand comm#ff850676 max_strand_ms 65.339 MED
kthread-strand comm#dca24885 max_strand_ms 61.5474 MED
kthread-strand comm#0dbd57b1 max_strand_ms 60.6359 MED
kthread-strand comm#0878f599 max_strand_ms 55.4664 MED
kthread-strand comm#180cd039 max_strand_ms 47.5361 MED
mid-trace-death comm#e3e5febe/283302 before_end_s 28.0189 MED
kthread-strand comm#2f099b5a max_strand_ms 45.6186 MED
kthread-strand comm#e1c397f1 max_strand_ms 42.5753 MED
kthread-strand comm#1beaeefc max_strand_ms 33.2128 MED
kthread-strand comm#12f62659 max_strand_ms 21.2363 MED
kthread-strand comm#fa4c2d9c max_strand_ms 20.6572 MED
mid-trace-death comm#e3e5febe/283389 before_end_s 7.00054 MED
CAPTURE complete -- 201875 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 89.1k wake2run; p50 2us p99 947us p999 948us worst 65028us; 95.6% fast(<100us) / 0.0% mid / 4.3% tick-floor(>=900us); 0.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~52907 distinct values; inversion ratio 0.70
LOCATED: 3 structured region(s) FEW_UNIQUE[17%..17%], FEW_UNIQUE[23%..24%], FEW_UNIQUE[29%..30%]
REPORT kstrand
VERDICT: 44 strands across 11 per-CPU kthreads; worst HELD strand 130.4ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#cfa27126 15 12 130.4 130.4 12 0 tid=0 0%
comm#ff850676 2 10 65.3 65.3 10 0 tid=0 0%
comm#dca24885 9 1 61.5 61.5 1 0 tid=0 0%
comm#0dbd57b1 7 4 60.6 60.6 4 0 tid=0 0%
comm#0878f599 0 7 55.5 55.5 7 0 tid=0 0%
comm#180cd039 5 4 47.5 47.5 4 0 tid=0 0%
comm#2f099b5a 2 1 45.6 45.6 1 0 tid=0 0%
comm#e1c397f1 8 1 42.6 42.6 1 0 tid=0 0%
comm#1beaeefc 4 1 33.2 33.2 1 0 tid=0 0%
comm#12f62659 6 2 21.2 21.2 2 0 tid=0 0%
comm#fa4c2d9c 3 1 20.7 20.7 1 0 tid=0 0%
WORKLOAD cold-wake-starve-16c-scx_flow
THERMAL/POWER
cpu temp peak 58.5 C avg 53.6 C
power avg 90.0 W peak 113.7 W
idle C2 avg 92.9% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
discord 23212ms profile_vs_mean 3.13559 HIGH
hot-cpu 15 l2_miss_share 36.4662 HIGH
mid-trace-death comm#e3e5febe/279738 before_end_s 35.0394 MED
long-range-dependent dispatch-rate hurst 0.923058 MED
mid-trace-death comm#e3e5febe/279767 before_end_s 28.0208 MED
mid-trace-death comm#e3e5febe/279854 before_end_s 7.01936 MED
CAPTURE complete -- 214084 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 96.5k wake2run; p50 1us p99 5us p999 45us worst 225us; 100.0% fast(<100us) / 0.0% mid / 0.0% tick-floor(>=900us); 0.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~58780 distinct values; inversion ratio 0.59
LOCATED: 3 structured region(s) FEW_UNIQUE[21%..24%], FEW_UNIQUE[25%..28%], FEW_UNIQUE[28%..35%]
WORKLOAD cold-wake-starve-16c-scx_lavd
THERMAL/POWER
cpu temp peak 60.2 C avg 54.6 C
power avg 89.1 W peak 113.9 W
idle C2 avg 92.3% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
hot-cpu 3 l2_miss_share 61.7665 HIGH
kthread-strand comm#0878f233 max_strand_ms 11.4087 MED
mid-trace-death comm#e3e5febe/279150 before_end_s 36.3 MED
long-range-dependent dispatch-rate hurst 0.900041 MED
discord 17014ms profile_vs_mean 2.78643 MED
kthread-strand comm#cfa26dc0 max_strand_ms 7.47724 MED
kthread-strand comm#fa4c32b5 max_strand_ms 7.00236 MED
kthread-strand comm#0357d927 max_strand_ms 5.99865 MED
mid-trace-death comm#e3e5febe/279179 before_end_s 29.297 MED
kthread-strand comm#1beaeefc max_strand_ms 5.99067 MED
kthread-strand comm#cfa26f73 max_strand_ms 5.02374 MED
kthread-strand comm#fa4c2d9c max_strand_ms 5.00267 MED
mid-trace-death comm#e3e5febe/279262 before_end_s 8.2851 MED
CAPTURE complete -- 193952 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 84.0k wake2run; p50 2us p99 4947us p999 5092us worst 12328us; 97.9% fast(<100us) / 0.1% mid / 1.9% tick-floor(>=900us); 0.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~56459 distinct values; inversion ratio 0.63
LOCATED: 2 structured region(s) FEW_UNIQUE[74%..75%], FEW_UNIQUE[91%..91%]
REPORT kstrand
VERDICT: 30 strands across 7 per-CPU kthreads; worst HELD strand 11.4ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#0878f233 0 3 11.4 11.4 3 0 tid=0 0%
comm#cfa26dc0 15 10 7.5 7.5 10 0 tid=0 0%
comm#fa4c32b5 3 6 7.0 7.0 6 0 tid=0 0%
comm#0357d927 1 8 6.0 6.0 8 0 tid=0 0%
comm#1beaeefc 4 1 6.0 6.0 1 0 tid=0 0%
comm#cfa26f73 15 1 5.0 5.0 1 0 tid=0 0%
comm#fa4c2d9c 3 1 5.0 5.0 1 0 tid=0 0%
WORKLOAD cold-wake-starve-16c-scx_p2dq
THERMAL/POWER
cpu temp peak 58.9 C avg 53.6 C
power avg 94.0 W peak 110.1 W
idle C2 avg 92.3% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
discord 15328ms profile_vs_mean 4.86813 HIGH
hot-cpu 15 l2_miss_share 33.3271 HIGH
kthread-strand comm#0878f233 max_strand_ms 20.9985 MED
mid-trace-death comm#e3e5febe/280920 before_end_s 35.0278 MED
long-range-dependent dispatch-rate hurst 0.918406 MED
kthread-strand comm#e1c397f1 max_strand_ms 20.997 MED
kthread-strand comm#cfa26f73 max_strand_ms 20.9961 MED
kthread-strand comm#0357d927 max_strand_ms 20.9833 MED
kthread-strand comm#ff850676 max_strand_ms 19.0022 MED
kthread-strand comm#2f099b5a max_strand_ms 19.0003 MED
mid-trace-death comm#e3e5febe/280949 before_end_s 28.0108 MED
kthread-strand comm#0878f599 max_strand_ms 18.0055 MED
kthread-strand comm#493ffe92 max_strand_ms 18.0002 MED
kthread-strand comm#e1c392d8 max_strand_ms 12.0046 MED
kthread-strand comm#fa4c2d9c max_strand_ms 11.5699 MED
kthread-strand comm#dca24885 max_strand_ms 9.00028 MED
mid-trace-death comm#e3e5febe/281036 before_end_s 7.00692 MED
CAPTURE complete -- 193655 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 83.2k wake2run; p50 2us p99 45us p999 20947us worst 23828us; 99.1% fast(<100us) / 0.0% mid / 0.9% tick-floor(>=900us); 0.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~49372 distinct values; inversion ratio 0.70
LOCATED: 8 structured region(s) NEARLY_SORTED[8%..9%], FEW_UNIQUE[10%..11%], FEW_UNIQUE[20%..21%], FEW_UNIQUE[22%..23%], FEW_UNIQUE[71%..71%] +3 more
REPORT kstrand
VERDICT: 84 strands across 11 per-CPU kthreads; worst HELD strand 21.0ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#0878f233 0 18 21.0 21.0 18 0 tid=0 0%
comm#e1c397f1 8 15 21.0 21.0 15 0 tid=0 0%
comm#cfa26f73 15 14 21.0 21.0 14 0 tid=0 0%
comm#0357d927 1 10 21.0 21.0 10 0 tid=0 0%
comm#ff850676 2 14 19.0 19.0 14 0 tid=0 0%
comm#2f099b5a 2 2 19.0 19.0 2 0 tid=0 0%
comm#0878f599 0 1 18.0 18.0 1 0 tid=0 0%
comm#493ffe92 10 1 18.0 18.0 1 0 tid=0 0%
comm#e1c392d8 8 1 12.0 12.0 1 0 tid=0 0%
comm#fa4c2d9c 3 7 11.6 11.6 7 0 tid=0 0%
comm#dca24885 9 1 9.0 9.0 1 0 tid=0 0%
WORKLOAD cold-wake-starve-16c-scx_rustland
THERMAL/POWER
cpu temp peak 59.1 C avg 53.9 C
power avg 90.7 W peak 109.1 W
idle C2 avg 92.3% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
hot-cpu 15 l2_miss_share 31.1002 HIGH
kthread-strand comm#1beaeefc max_strand_ms 21.9911 MED
mid-trace-death comm#e3e5febe/280339 before_end_s 37.878 MED
long-range-dependent dispatch-rate hurst 0.913196 MED
discord 2790ms profile_vs_mean 2.99867 MED
mid-trace-death comm#e3e5febe/280367 before_end_s 30.86 MED
mid-trace-death comm#e3e5febe/280456 before_end_s 9.85226 MED
mid-trace-death comm#e3e5febe/280486 before_end_s 2.83231 MED
CAPTURE complete -- 205007 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 89.5k wake2run; p50 2us p99 947us p999 1947us worst 20703us; 95.3% fast(<100us) / 0.3% mid / 4.4% tick-floor(>=900us); 0.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~58738 distinct values; inversion ratio 0.66
REPORT kstrand
VERDICT: 1 strands across 1 per-CPU kthreads; worst HELD strand 22.0ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#1beaeefc 4 1 22.0 22.0 1 0 tid=0 0%
WORKLOAD cold-wake-starve-16c-scx_rusty
THERMAL/POWER
cpu temp peak 58.8 C avg 53.6 C
power avg 91.5 W peak 110.4 W
idle C2 avg 92.6% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
hot-cpu 15 l2_miss_share 38.2915 HIGH
kthread-strand comm#cfa26dc0 max_strand_ms 21.0066 MED
mid-trace-death comm#e3e5febe/278563 before_end_s 35.046 MED
long-range-dependent dispatch-rate hurst 0.914365 MED
discord 10073ms profile_vs_mean 2.98377 MED
kthread-strand comm#ff850676 max_strand_ms 19.0016 MED
kthread-strand comm#0357d927 max_strand_ms 18.9996 MED
kthread-strand comm#493ffe92 max_strand_ms 17.988 MED
kthread-strand comm#d1030141 max_strand_ms 16.003 MED
kthread-strand comm#fa4c32b5 max_strand_ms 16.0007 MED
kthread-strand comm#e1c397f1 max_strand_ms 16.0006 MED
mid-trace-death comm#e3e5febe/278592 before_end_s 28.032 MED
kthread-strand comm#2f099b5a max_strand_ms 15.9975 MED
kthread-strand comm#cfa26f73 max_strand_ms 13.6452 MED
kthread-strand comm#dca24885 max_strand_ms 13.0066 MED
kthread-strand comm#12f62659 max_strand_ms 13.0008 MED
kthread-strand comm#ca8154b4 max_strand_ms 12.0056 MED
kthread-strand comm#0dbd57b1 max_strand_ms 7.99197 MED
mid-trace-death comm#e3e5febe/278678 before_end_s 7.01237 MED
CAPTURE complete -- 192769 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 82.6k wake2run; p50 2us p99 41us p999 19947us worst 41941us; 99.1% fast(<100us) / 0.0% mid / 0.9% tick-floor(>=900us); 0.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~50329 distinct values; inversion ratio 0.67
LOCATED: 6 structured region(s) NEARLY_SORTED[8%..9%], FEW_UNIQUE[13%..14%], FEW_UNIQUE[15%..16%], FEW_UNIQUE[17%..18%], FEW_UNIQUE[19%..19%] +1 more
REPORT kstrand
VERDICT: 65 strands across 13 per-CPU kthreads; worst HELD strand 21.0ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#cfa26dc0 15 13 21.0 21.0 13 0 tid=0 0%
comm#ff850676 2 7 19.0 19.0 7 0 tid=0 0%
comm#0357d927 1 27 19.0 19.0 27 0 tid=0 0%
comm#493ffe92 10 3 18.0 18.0 3 0 tid=0 0%
comm#d1030141 15 2 16.0 16.0 2 0 tid=0 0%
comm#fa4c32b5 3 2 16.0 16.0 2 0 tid=0 0%
comm#e1c397f1 8 2 16.0 16.0 2 0 tid=0 0%
comm#2f099b5a 2 1 16.0 16.0 1 0 tid=0 0%
comm#cfa26f73 15 2 13.6 13.6 2 0 tid=0 0%
comm#dca24885 9 2 13.0 13.0 2 0 tid=0 0%
comm#12f62659 6 2 13.0 13.0 2 0 tid=0 0%
comm#ca8154b4 14 1 12.0 12.0 1 0 tid=0 0%
comm#0dbd57b1 7 1 8.0 8.0 1 0 tid=0 0%
WORKLOAD cold-wake-starve-16c-scx_tickless
THERMAL/POWER
cpu temp peak 59.2 C avg 53.3 C
power avg 92.1 W peak 114.9 W
idle C2 avg 90.8% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
discord 14037ms profile_vs_mean 3.64281 HIGH
hot-cpu 0 l2_miss_share 29.9094 HIGH
kthread-strand comm#cfa27126 max_strand_ms 23.9387 MED
mid-trace-death comm#e3e5febe/281506 before_end_s 35.1061 MED
long-range-dependent dispatch-rate hurst 0.907525 MED
kthread-strand comm#0878f233 max_strand_ms 21.0041 MED
kthread-strand comm#ff850676 max_strand_ms 21.0011 MED
kthread-strand comm#0357d927 max_strand_ms 20.9999 MED
mid-trace-death comm#e3e5febe/281539 before_end_s 28.0831 MED
kthread-strand comm#fa4c2d9c max_strand_ms 20.9994 MED
kthread-strand comm#2f099b5a max_strand_ms 12.0026 MED
kthread-strand comm#cfa26f73 max_strand_ms 6.95845 MED
kthread-strand comm#d1030141 max_strand_ms 6.89961 MED
mid-trace-death comm#e3e5febe/281622 before_end_s 7.01209 MED
CAPTURE complete -- 179384 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 77.9k wake2run; p50 1us p99 95us p999 20951us worst 21947us; 99.0% fast(<100us) / 0.1% mid / 0.9% tick-floor(>=900us); 0.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~43813 distinct values; inversion ratio 0.59
LOCATED: 1 structured region(s) NEARLY_SORTED[9%..9%]
REPORT kstrand
VERDICT: 119 strands across 8 per-CPU kthreads; worst HELD strand 23.9ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#cfa27126 15 16 23.9 23.9 16 0 tid=0 0%
comm#0878f233 0 57 21.0 21.0 57 0 tid=0 0%
comm#ff850676 2 24 21.0 21.0 24 0 tid=0 0%
comm#0357d927 1 14 21.0 21.0 14 0 tid=0 0%
comm#fa4c2d9c 3 4 21.0 21.0 4 0 tid=0 0%
comm#2f099b5a 2 2 12.0 12.0 2 0 tid=0 0%
comm#cfa26f73 15 1 7.0 7.0 1 0 tid=0 0%
comm#d1030141 15 1 6.9 6.9 1 0 tid=0 0%
WORKLOAD fork-thread-EEVDF
THERMAL/POWER
cpu temp peak 61.8 C avg 52.4 C
power avg 104.0 W peak 166.5 W
idle C2 avg 74.7% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
order-starved tid=1250 passover_picks 3101 HIGH
held-cpu tid=151866 held_ms 9.56058 HIGH
self-preempt tid=1717 stint_wake_ratio 7962 HIGH
self-preempt tid=1650 stint_wake_ratio 7930 HIGH
self-preempt tid=1652 stint_wake_ratio 7922 HIGH
self-preempt tid=1827 stint_wake_ratio 7860 HIGH
order-starved tid=152140 passover_picks 2276 HIGH
held-cpu tid=151903 held_ms 7.93977 HIGH
self-preempt tid=1649 stint_wake_ratio 7858 HIGH
self-preempt tid=1651 stint_wake_ratio 7840 HIGH
self-preempt tid=1826 stint_wake_ratio 7837 HIGH
order-starved tid=152219 passover_picks 2146 HIGH
held-cpu tid=151899 held_ms 7.76521 HIGH
self-preempt tid=1718 stint_wake_ratio 7833 HIGH
hot-waker 0 wakes_issued 1417 MED
long-range-dependent dispatch-rate hurst 0.877921 MED
migration-avalanche migration-rate runs 73 MED
hot-waker 151983 wakes_issued 1161 MED
hot-waker 151908 wakes_issued 908 MED
long-range-dependent migration-rate hurst 0.662987 MED
hot-cpu 0 l2_miss_share 7.01477 LOW
CAPTURE complete -- 667107 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 66.3k wake2run; p50 303us p99 40860us p999 87712us worst 138354us; 41.5% fast(<100us) / 12.0% mid / 46.5% tick-floor(>=900us); 1.6% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~65256 distinct values; inversion ratio 0.66
LOCATED: 1 structured region(s) NEARLY_SORTED[8%..9%]
REPORT dispatch-stall
VERDICT: 30.8k saturated wakes over the 900us floor; PREEMPT-STARVED 1% (0 intervening picks) / ORDER-STARVED 99% (CPU served others first); avg 52.0 pass-overs, p99 320 pass-overs
WORKLOAD fork-thread-PANDEMONIUM-ADAPTIVE
THERMAL/POWER
cpu temp peak 61.5 C avg 55.9 C
power avg 102.2 W peak 165.7 W
idle C2 avg 76.6% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
order-starved tid=801 passover_picks 1020 HIGH
held-cpu tid=154154 held_ms 1.15451 HIGH
self-preempt tid=1826 stint_wake_ratio 8627 HIGH
self-preempt tid=1827 stint_wake_ratio 8178 HIGH
self-preempt tid=1652 stint_wake_ratio 8165 HIGH
self-preempt tid=1649 stint_wake_ratio 8148 HIGH
order-starved tid=154849 passover_picks 748 HIGH
held-cpu tid=154151 held_ms 1.10273 HIGH
self-preempt tid=1650 stint_wake_ratio 8127 HIGH
self-preempt tid=1651 stint_wake_ratio 8122 HIGH
self-preempt tid=1718 stint_wake_ratio 8117 HIGH
order-starved tid=154447 passover_picks 736 HIGH
held-cpu tid=154195 held_ms 1.04007 HIGH
self-preempt tid=1717 stint_wake_ratio 8104 HIGH
hot-waker 154116 wakes_issued 2946 MED
long-range-dependent dispatch-rate hurst 0.858844 MED
migration-avalanche migration-rate runs 214 MED
discord 6078ms profile_vs_mean 1.51418 MED
hot-waker 154281 wakes_issued 2831 MED
hot-waker 154265 wakes_issued 2773 MED
long-range-dependent migration-rate hurst 0.837607 MED
hot-cpu 2 l2_miss_share 7.04061 LOW
CAPTURE complete -- 2208187 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 255k wake2run; p50 283us p99 5030us p999 10032us worst 17922us; 31.9% fast(<100us) / 43.6% mid / 24.5% tick-floor(>=900us); 0.7% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~254654 distinct values; inversion ratio 0.52
LOCATED: 14 structured region(s) NEARLY_SORTED[0%..0%], NEARLY_SORTED[8%..8%], NEARLY_SORTED[9%..9%], NEARLY_SORTED[9%..9%], NEARLY_SORTED[13%..13%] +9 more
REPORT dispatch-stall
VERDICT: 62.3k saturated wakes over the 900us floor; PREEMPT-STARVED 0% (0 intervening picks) / ORDER-STARVED 100% (CPU served others first); avg 43.9 pass-overs, p99 162 pass-overs
WORKLOAD fork-thread-PANDEMONIUM-BPF
THERMAL/POWER
cpu temp peak 61.0 C avg 54.8 C
power avg 100.7 W peak 165.3 W
idle C2 avg 75.7% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
order-starved tid=153146 passover_picks 1194 HIGH
held-cpu tid=153128 held_ms 0.99191 HIGH
self-preempt tid=1826 stint_wake_ratio 8665 HIGH
self-preempt tid=1652 stint_wake_ratio 8481 HIGH
self-preempt tid=1718 stint_wake_ratio 8047 HIGH
self-preempt tid=1651 stint_wake_ratio 8035 HIGH
order-starved tid=153386 passover_picks 1106 HIGH
held-cpu tid=152998 held_ms 0.98602 HIGH
self-preempt tid=1717 stint_wake_ratio 8018 HIGH
self-preempt tid=1827 stint_wake_ratio 8012 HIGH
self-preempt tid=1649 stint_wake_ratio 8011 HIGH
order-starved tid=153587 passover_picks 979 HIGH
held-cpu tid=153167 held_ms 0.93923 HIGH
self-preempt tid=1650 stint_wake_ratio 8005 HIGH
long-range-dependent migration-rate hurst 0.848329 MED
migration-avalanche migration-rate runs 81 MED
discord 3832ms profile_vs_mean 1.93348 MED
long-range-dependent dispatch-rate hurst 0.836737 MED
hot-cpu 2 l2_miss_share 6.895 LOW
CAPTURE complete -- 2059452 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 234k wake2run; p50 238us p99 4897us p999 11108us worst 17941us; 35.8% fast(<100us) / 40.1% mid / 24.0% tick-floor(>=900us); 0.3% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~233739 distinct values; inversion ratio 0.61
LOCATED: 8 structured region(s) NEARLY_SORTED[0%..0%], NEARLY_SORTED[33%..33%], NEARLY_SORTED[46%..47%], NEARLY_SORTED[79%..80%], NEARLY_SORTED[86%..86%] +3 more
REPORT dispatch-stall
VERDICT: 56.2k saturated wakes over the 900us floor; PREEMPT-STARVED 0% (0 intervening picks) / ORDER-STARVED 100% (CPU served others first); avg 47.5 pass-overs, p99 169 pass-overs
WORKLOAD fork-thread-scx_beerland
THERMAL/POWER
cpu temp peak 61.0 C avg 57.1 C
power avg 102.9 W peak 160.2 W
idle C2 avg 75.6% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
order-starved tid=16 passover_picks 8306 HIGH
held-cpu tid=167435 held_ms 1.54156 HIGH
self-preempt tid=1826 stint_wake_ratio 8610 HIGH
self-preempt tid=1651 stint_wake_ratio 8609 HIGH
self-preempt tid=1652 stint_wake_ratio 8604 HIGH
self-preempt tid=1717 stint_wake_ratio 8580 HIGH
order-starved tid=167621 passover_picks 5264 HIGH
held-cpu tid=167722 held_ms 1.38682 HIGH
self-preempt tid=1649 stint_wake_ratio 8565 HIGH
self-preempt tid=1827 stint_wake_ratio 8561 HIGH
self-preempt tid=1650 stint_wake_ratio 8558 HIGH
order-starved tid=124560 passover_picks 2873 HIGH
held-cpu tid=167437 held_ms 1.12054 HIGH
self-preempt tid=1718 stint_wake_ratio 8554 HIGH
long-range-dependent dispatch-rate hurst 0.920251 MED
migration-avalanche migration-rate runs 1871 MED
discord 3924ms profile_vs_mean 1.99747 MED
hot-cpu 15 l2_miss_share 11.0496 MED
long-range-dependent migration-rate hurst 0.885249 MED
CAPTURE complete -- 461537 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 39.0k wake2run; p50 19387us p99 31575us p999 51276us worst 97279us; 30.6% fast(<100us) / 3.4% mid / 66.0% tick-floor(>=900us); 36.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~38977 distinct values; inversion ratio 0.51
LOCATED: 7 structured region(s) NEARLY_SORTED[30%..32%], NEARLY_SORTED[33%..34%], NEARLY_SORTED[60%..62%], NEARLY_SORTED[74%..75%], NEARLY_SORTED[76%..78%] +2 more
REPORT dispatch-stall
VERDICT: 25.7k saturated wakes over the 900us floor; PREEMPT-STARVED 0% (0 intervening picks) / ORDER-STARVED 100% (CPU served others first); avg 62.3 pass-overs, p99 470 pass-overs
WORKLOAD fork-thread-scx_bpfland
THERMAL/POWER
cpu temp peak 60.0 C avg 55.6 C
power avg 86.4 W peak 157.6 W
idle C2 avg 88.2% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
order-starved tid=16 passover_picks 4893 HIGH
held-cpu tid=155343 held_ms 44.0516 HIGH
hot-waker 155546 wakes_issued 3702 HIGH
self-preempt tid=1652 stint_wake_ratio 8033 HIGH
self-preempt tid=1651 stint_wake_ratio 8033 HIGH
self-preempt tid=1649 stint_wake_ratio 8030 HIGH
self-preempt tid=1826 stint_wake_ratio 8015 HIGH
order-starved tid=156149 passover_picks 792 HIGH
held-cpu tid=155335 held_ms 37.0703 HIGH
hot-waker 0 wakes_issued 3290 HIGH
self-preempt tid=1650 stint_wake_ratio 8012 HIGH
self-preempt tid=1827 stint_wake_ratio 8004 HIGH
self-preempt tid=1718 stint_wake_ratio 8001 HIGH
order-starved tid=155605 passover_picks 647 HIGH
held-cpu tid=155536 held_ms 9.45679 HIGH
hot-waker 155414 wakes_issued 2458 HIGH
self-preempt tid=1717 stint_wake_ratio 7984 HIGH
long-range-dependent dispatch-rate hurst 1.02563 MED
migration-avalanche migration-rate runs 2586 MED
discord 1886ms profile_vs_mean 1.69339 MED
hot-cpu 15 l2_miss_share 9.89641 MED
long-range-dependent migration-rate hurst 1.00678 MED
CAPTURE complete -- 919648 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 128k wake2run; p50 8us p99 16247us p999 155231us worst 206730us; 69.7% fast(<100us) / 5.1% mid / 25.1% tick-floor(>=900us); 47.8% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~128112 distinct values; inversion ratio 0.54
LOCATED: 1 structured region(s) NEARLY_SORTED[35%..35%]
REPORT dispatch-stall
VERDICT: 32.2k saturated wakes over the 900us floor; PREEMPT-STARVED 0% (0 intervening picks) / ORDER-STARVED 100% (CPU served others first); avg 68.1 pass-overs, p99 835 pass-overs
WORKLOAD fork-thread-scx_cake
THERMAL/POWER
cpu temp peak 60.8 C avg 57.3 C
power avg 101.1 W peak 160.6 W
idle C2 avg 75.6% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
order-starved tid=165045 passover_picks 1542 HIGH
held-cpu tid=165232 held_ms 3.45696 HIGH
self-preempt tid=1652 stint_wake_ratio 8665 HIGH
self-preempt tid=1651 stint_wake_ratio 8652 HIGH
self-preempt tid=1717 stint_wake_ratio 8650 HIGH
self-preempt tid=1650 stint_wake_ratio 8632 HIGH
order-starved tid=16 passover_picks 1235 HIGH
held-cpu tid=165222 held_ms 3.24371 HIGH
self-preempt tid=1649 stint_wake_ratio 8625 HIGH
self-preempt tid=1827 stint_wake_ratio 8605 HIGH
self-preempt tid=1718 stint_wake_ratio 8570 HIGH
order-starved tid=165038 passover_picks 1103 HIGH
held-cpu tid=165146 held_ms 3.019 HIGH
self-preempt tid=1826 stint_wake_ratio 8539 HIGH
hot-waker 165179 wakes_issued 1529 MED
long-range-dependent dispatch-rate hurst 0.82535 MED
migration-avalanche migration-rate runs 35 MED
discord 3201ms profile_vs_mean 1.98307 MED
hot-waker 165032 wakes_issued 1324 MED
hot-waker 164991 wakes_issued 1262 MED
long-range-dependent migration-rate hurst 0.635057 MED
hot-cpu 15 l2_miss_share 6.62036 LOW
CAPTURE complete -- 1114622 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 149k wake2run; p50 21us p99 27932us p999 63136us worst 204152us; 65.9% fast(<100us) / 15.3% mid / 18.8% tick-floor(>=900us); 0.1% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~149275 distinct values; inversion ratio 0.63
REPORT dispatch-stall
VERDICT: 28.1k saturated wakes over the 900us floor; PREEMPT-STARVED 0% (0 intervening picks) / ORDER-STARVED 100% (CPU served others first); avg 86.8 pass-overs, p99 646 pass-overs
WORKLOAD fork-thread-scx_cosmos
THERMAL/POWER
cpu temp peak 59.0 C avg 54.5 C
power avg 104.8 W peak 155.3 W
idle C2 avg 69.8% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
order-starved tid=163792 passover_picks 17930 HIGH
held-cpu tid=163688 held_ms 41.0078 HIGH
hot-waker 163767 wakes_issued 3746 HIGH
self-preempt tid=164473 stint_wake_ratio 15531 HIGH
self-preempt tid=164557 stint_wake_ratio 15450 HIGH
self-preempt tid=164479 stint_wake_ratio 15438 HIGH
self-preempt tid=164434 stint_wake_ratio 15370 HIGH
order-starved tid=164436 passover_picks 14565 HIGH
held-cpu tid=163655 held_ms 40.2614 HIGH
self-preempt tid=164480 stint_wake_ratio 15193 HIGH
self-preempt tid=164436 stint_wake_ratio 15107 HIGH
self-preempt tid=164555 stint_wake_ratio 15056 HIGH
order-starved tid=164434 passover_picks 14313 HIGH
held-cpu tid=163696 held_ms 39.2932 HIGH
self-preempt tid=164471 stint_wake_ratio 15037 HIGH
long-range-dependent dispatch-rate hurst 0.944445 MED
migration-avalanche migration-rate runs 4495 MED
discord 5067ms profile_vs_mean 1.5287 MED
long-range-dependent migration-rate hurst 0.925112 MED
hot-cpu 3 l2_miss_share 6.60398 LOW
CAPTURE complete -- 4031780 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 368k wake2run; p50 8us p99 7164us p999 506012us worst 656598us; 81.0% fast(<100us) / 7.8% mid / 11.2% tick-floor(>=900us); 48.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~367624 distinct values; inversion ratio 0.43
LOCATED: 5 structured region(s) NEARLY_SORTED[6%..7%], NEARLY_SORTED[8%..9%], NEARLY_SORTED[20%..20%], NEARLY_SORTED[65%..65%], NEARLY_SORTED[67%..67%]
REPORT dispatch-stall
VERDICT: 41.1k saturated wakes over the 900us floor; PREEMPT-STARVED 5% (0 intervening picks) / ORDER-STARVED 95% (CPU served others first); avg 591.0 pass-overs, p99 18858 pass-overs
WORKLOAD fork-thread-scx_flash
THERMAL/POWER
cpu temp peak 60.0 C avg 55.9 C
power avg 111.3 W peak 156.5 W
idle C2 avg 59.9% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
order-starved tid=166918 passover_picks 14260 HIGH
held-cpu tid=166213 held_ms 9.1106 HIGH
hot-waker 0 wakes_issued 7845 HIGH
self-preempt tid=166918 stint_wake_ratio 10194 HIGH
self-preempt tid=166917 stint_wake_ratio 9953 HIGH
self-preempt tid=166915 stint_wake_ratio 9808 HIGH
self-preempt tid=166916 stint_wake_ratio 9745 HIGH
order-starved tid=166914 passover_picks 11628 HIGH
held-cpu tid=166216 held_ms 7.39008 HIGH
hot-waker 166097 wakes_issued 5625 HIGH
self-preempt tid=166914 stint_wake_ratio 9719 HIGH
self-preempt tid=166906 stint_wake_ratio 9616 HIGH
self-preempt tid=1718 stint_wake_ratio 9574 HIGH
order-starved tid=166915 passover_picks 9624 HIGH
held-cpu tid=166208 held_ms 5.63372 HIGH
hot-waker 166290 wakes_issued 5435 HIGH
self-preempt tid=1826 stint_wake_ratio 9573 HIGH
long-range-dependent dispatch-rate hurst 0.941542 MED
migration-avalanche migration-rate runs 4013 MED
discord 557ms profile_vs_mean 1.67566 MED
long-range-dependent migration-rate hurst 0.914058 MED
hot-cpu 7 l2_miss_share 6.56767 LOW
CAPTURE complete -- 2892445 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 527k wake2run; p50 5us p99 3030us p999 207873us worst 721887us; 87.5% fast(<100us) / 8.9% mid / 3.6% tick-floor(>=900us); 47.4% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~527029 distinct values; inversion ratio 0.55
LOCATED: 2 structured region(s) NEARLY_SORTED[53%..53%], NEARLY_SORTED[69%..69%]
REPORT dispatch-stall
VERDICT: 18.8k saturated wakes over the 900us floor; PREEMPT-STARVED 1% (0 intervening picks) / ORDER-STARVED 99% (CPU served others first); avg 838.2 pass-overs, p99 7712 pass-overs
WORKLOAD fork-thread-scx_flow
THERMAL/POWER
cpu temp peak 61.5 C avg 58.3 C
power avg 108.5 W peak 165.6 W
idle C2 avg 68.9% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
order-starved tid=159890 passover_picks 5541 HIGH
held-cpu tid=159271 held_ms 1.03604 HIGH
self-preempt tid=160005 stint_wake_ratio 15657 HIGH
self-preempt tid=159448 stint_wake_ratio 13985 HIGH
self-preempt tid=159832 stint_wake_ratio 13200 HIGH
self-preempt tid=159327 stint_wake_ratio 13075 HIGH
order-starved tid=159636 passover_picks 5377 HIGH
held-cpu tid=159098 held_ms 1.00184 HIGH
self-preempt tid=159995 stint_wake_ratio 12617 HIGH
self-preempt tid=159793 stint_wake_ratio 12468 HIGH
self-preempt tid=159365 stint_wake_ratio 12220 HIGH
order-starved tid=159314 passover_picks 5307 HIGH
held-cpu tid=159217 held_ms 0.98508 HIGH
self-preempt tid=159530 stint_wake_ratio 12075 HIGH
hot-waker 159307 wakes_issued 6611 MED
long-range-dependent migration-rate hurst 1.00497 MED
migration-avalanche migration-rate runs 1208 MED
discord 4005ms profile_vs_mean 1.66464 MED
hot-waker 159231 wakes_issued 5993 MED
long-range-dependent dispatch-rate hurst 0.881101 MED
hot-cpu 4 l2_miss_share 6.80399 LOW
CAPTURE complete -- 6223434 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 835k wake2run; p50 2us p99 2500us p999 5991us worst 67857us; 86.5% fast(<100us) / 9.3% mid / 4.2% tick-floor(>=900us); 2.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~743702 distinct values; inversion ratio 0.44
REPORT dispatch-stall
VERDICT: 35.1k saturated wakes over the 900us floor; PREEMPT-STARVED 0% (0 intervening picks) / ORDER-STARVED 100% (CPU served others first); avg 115.3 pass-overs, p99 233 pass-overs
WORKLOAD fork-thread-scx_lavd
THERMAL/POWER
cpu temp peak 59.8 C avg 57.4 C
power avg 109.4 W peak 159.2 W
idle C2 avg 68.2% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
order-starved tid=158212 passover_picks 15858 HIGH
held-cpu tid=157866 held_ms 8.67099 HIGH
hot-waker 157918 wakes_issued 6733 HIGH
self-preempt tid=158337 stint_wake_ratio 17835 HIGH
self-preempt tid=158174 stint_wake_ratio 17794 HIGH
self-preempt tid=158133 stint_wake_ratio 17627 HIGH
self-preempt tid=158538 stint_wake_ratio 17556 HIGH
order-starved tid=158289 passover_picks 14122 HIGH
held-cpu tid=157835 held_ms 5.65895 HIGH
hot-waker 157792 wakes_issued 6427 HIGH
self-preempt tid=158212 stint_wake_ratio 17294 HIGH
self-preempt tid=158422 stint_wake_ratio 17267 HIGH
self-preempt tid=158424 stint_wake_ratio 17058 HIGH
order-starved tid=158337 passover_picks 13277 HIGH
held-cpu tid=158204 held_ms 5.64288 HIGH
hot-waker 157790 wakes_issued 6246 HIGH
self-preempt tid=158251 stint_wake_ratio 16885 HIGH
long-range-dependent dispatch-rate hurst 0.897921 MED
migration-avalanche migration-rate runs 2352 MED
discord 6245ms profile_vs_mean 1.52576 MED
long-range-dependent migration-rate hurst 0.717947 MED
hot-cpu 2 l2_miss_share 6.42729 LOW
CAPTURE complete -- 3864827 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 606k wake2run; p50 8us p99 17712us p999 42353us worst 265176us; 85.8% fast(<100us) / 7.5% mid / 6.7% tick-floor(>=900us); 5.2% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~596800 distinct values; inversion ratio 0.55
REPORT dispatch-stall
VERDICT: 40.7k saturated wakes over the 900us floor; PREEMPT-STARVED 1% (0 intervening picks) / ORDER-STARVED 99% (CPU served others first); avg 323.2 pass-overs, p99 1881 pass-overs
WORKLOAD fork-thread-scx_p2dq
THERMAL/POWER
cpu temp peak 60.5 C avg 54.3 C
power avg 103.7 W peak 159.8 W
idle C2 avg 77.3% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
order-starved tid=1030 passover_picks 336 HIGH
held-cpu tid=161483 held_ms 2.9356 HIGH
self-preempt tid=1717 stint_wake_ratio 8516 HIGH
self-preempt tid=1826 stint_wake_ratio 8514 HIGH
self-preempt tid=1650 stint_wake_ratio 8502 HIGH
self-preempt tid=1827 stint_wake_ratio 8448 HIGH
order-starved tid=161522 passover_picks 216 HIGH
held-cpu tid=161677 held_ms 2.84425 HIGH
self-preempt tid=1649 stint_wake_ratio 8448 HIGH
self-preempt tid=1651 stint_wake_ratio 8426 HIGH
self-preempt tid=1652 stint_wake_ratio 8377 HIGH
order-starved tid=161949 passover_picks 140 HIGH
held-cpu tid=161629 held_ms 2.67056 HIGH
self-preempt tid=1718 stint_wake_ratio 8248 HIGH
long-range-dependent dispatch-rate hurst 0.781316 MED
migration-avalanche migration-rate runs 3769 MED
discord 3107ms profile_vs_mean 1.84357 MED
long-range-dependent migration-rate hurst 0.613259 MED
hot-cpu 3 l2_miss_share 7.54885 LOW
CAPTURE complete -- 929704 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 85.1k wake2run; p50 6us p99 34473us p999 35588us worst 36930us; 61.5% fast(<100us) / 10.8% mid / 27.7% tick-floor(>=900us); 25.9% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~82450 distinct values; inversion ratio 0.46
REPORT dispatch-stall
VERDICT: 23.6k saturated wakes over the 900us floor; PREEMPT-STARVED 0% (0 intervening picks) / ORDER-STARVED 100% (CPU served others first); avg 92.8 pass-overs, p99 131 pass-overs
WORKLOAD fork-thread-scx_rustland
THERMAL/POWER
cpu temp peak 60.2 C avg 56.9 C
power avg 95.5 W peak 158.1 W
idle C2 avg 84.3% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
order-starved tid=160344 passover_picks 258435 HIGH
held-cpu tid=160488 held_ms 3.3725 HIGH
self-preempt tid=160344 stint_wake_ratio 38450 HIGH
self-preempt tid=1826 stint_wake_ratio 6338 HIGH
self-preempt tid=1649 stint_wake_ratio 6337 HIGH
self-preempt tid=1717 stint_wake_ratio 6337 HIGH
order-starved tid=16 passover_picks 529 HIGH
held-cpu tid=160405 held_ms 2.94509 HIGH
self-preempt tid=1650 stint_wake_ratio 6337 HIGH
self-preempt tid=1652 stint_wake_ratio 6336 HIGH
self-preempt tid=1651 stint_wake_ratio 6336 HIGH
order-starved tid=124560 passover_picks 71 HIGH
held-cpu tid=160406 held_ms 2.76263 HIGH
self-preempt tid=1827 stint_wake_ratio 6336 HIGH
migration-avalanche migration-rate runs 2311 MED
discord 4006ms profile_vs_mean 1.95658 MED
hot-cpu 15 l2_miss_share 16.5039 MED
CAPTURE complete -- 411736 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 27.7k wake2run; p50 17743us p99 21477us p999 22912us worst 23421us; 5.9% fast(<100us) / 3.6% mid / 90.5% tick-floor(>=900us); 45.3% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~27692 distinct values; inversion ratio 0.78
LOCATED: 1 structured region(s) NEARLY_SORTED[44%..46%]
REPORT dispatch-stall
VERDICT: 25.1k saturated wakes over the 900us floor; PREEMPT-STARVED 0% (0 intervening picks) / ORDER-STARVED 100% (CPU served others first); avg 49.1 pass-overs, p99 78 pass-overs
WORKLOAD fork-thread-scx_rusty
THERMAL/POWER
cpu temp peak 60.8 C avg 56.2 C
power avg 110.5 W peak 162.8 W
idle C2 avg 64.5% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
order-starved tid=157236 passover_picks 1745 HIGH
held-cpu tid=156696 held_ms 15.0317 HIGH
hot-waker 156493 wakes_issued 6088 HIGH
self-preempt tid=1651 stint_wake_ratio 8696 HIGH
self-preempt tid=1649 stint_wake_ratio 8507 HIGH
self-preempt tid=1718 stint_wake_ratio 8428 HIGH
self-preempt tid=1827 stint_wake_ratio 8399 HIGH
order-starved tid=156905 passover_picks 1694 HIGH
held-cpu tid=156524 held_ms 14.034 HIGH
self-preempt tid=1650 stint_wake_ratio 7623 HIGH
self-preempt tid=1826 stint_wake_ratio 7564 HIGH
self-preempt tid=1652 stint_wake_ratio 7562 HIGH
order-starved tid=156508 passover_picks 1562 HIGH
held-cpu tid=156493 held_ms 13.3733 HIGH
self-preempt tid=1717 stint_wake_ratio 7557 HIGH
long-range-dependent dispatch-rate hurst 0.990095 MED
migration-avalanche migration-rate runs 390 MED
discord 696ms profile_vs_mean 1.64171 MED
long-range-dependent migration-rate hurst 0.788108 MED
hot-cpu 2 l2_miss_share 7.24332 LOW
CAPTURE complete -- 2487678 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 396k wake2run; p50 5us p99 34297us p999 82003us worst 441614us; 85.4% fast(<100us) / 7.2% mid / 7.4% tick-floor(>=900us); 0.8% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~370887 distinct values; inversion ratio 0.52
LOCATED: 7 structured region(s) NEARLY_SORTED[66%..66%], NEARLY_SORTED[68%..69%], NEARLY_SORTED[69%..69%], NEARLY_SORTED[71%..71%], NEARLY_SORTED[72%..73%] +2 more
REPORT dispatch-stall
VERDICT: 29.3k saturated wakes over the 900us floor; PREEMPT-STARVED 1% (0 intervening picks) / ORDER-STARVED 99% (CPU served others first); avg 246.2 pass-overs, p99 4535 pass-overs
WORKLOAD fork-thread-scx_tickless
THERMAL/POWER
cpu temp peak 61.8 C avg 56.4 C
power avg 105.2 W peak 164.0 W
idle C2 avg 70.3% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
order-starved tid=16 passover_picks 5299 HIGH
held-cpu tid=801 held_ms 14.7649 HIGH
self-preempt tid=1827 stint_wake_ratio 6516 HIGH
self-preempt tid=1649 stint_wake_ratio 6512 HIGH
self-preempt tid=1718 stint_wake_ratio 6511 HIGH
self-preempt tid=1651 stint_wake_ratio 6510 HIGH
order-starved tid=124560 passover_picks 699 HIGH
held-cpu tid=162736 held_ms 1.51026 HIGH
self-preempt tid=1826 stint_wake_ratio 6509 HIGH
self-preempt tid=1717 stint_wake_ratio 6509 HIGH
self-preempt tid=1652 stint_wake_ratio 6508 HIGH
order-starved tid=1651 passover_picks 465 HIGH
held-cpu tid=162664 held_ms 1.45042 HIGH
self-preempt tid=1650 stint_wake_ratio 6506 HIGH
long-range-dependent dispatch-rate hurst 0.759586 MED
migration-avalanche migration-rate runs 2186 MED
discord 8232ms profile_vs_mean 1.67735 MED
long-range-dependent migration-rate hurst 0.756083 MED
hot-cpu 8 l2_miss_share 8.01603 LOW
CAPTURE complete -- 391779 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 43.3k wake2run; p50 13628us p99 38073us p999 69913us worst 125455us; 36.9% fast(<100us) / 3.5% mid / 59.5% tick-floor(>=900us); 47.3% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~41257 distinct values; inversion ratio 0.79
REPORT dispatch-stall
VERDICT: 25.8k saturated wakes over the 900us floor; PREEMPT-STARVED 0% (0 intervening picks) / ORDER-STARVED 100% (CPU served others first); avg 44.6 pass-overs, p99 173 pass-overs
WORKLOAD ipc-EEVDF-16c
THERMAL/POWER
cpu temp peak 58.8 C avg 52.5 C
power avg 108.8 W peak 141.3 W
idle C2 avg 46.3% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
held-cpu comm#8a5f93b2 held_ms 165.725 HIGH
self-preempt tid=250801 stint_wake_ratio 26186 HIGH
self-preempt tid=250802 stint_wake_ratio 24158 HIGH
self-preempt tid=250796 stint_wake_ratio 18689 HIGH
self-preempt tid=250805 stint_wake_ratio 18525 HIGH
held-cpu comm#8a5f93b2 held_ms 146.339 HIGH
self-preempt tid=250795 stint_wake_ratio 16783 HIGH
self-preempt tid=250792 stint_wake_ratio 13011 HIGH
self-preempt tid=250804 stint_wake_ratio 11865 HIGH
held-cpu comm#8a5f93b2 held_ms 135.383 HIGH
self-preempt tid=250786 stint_wake_ratio 10193 HIGH
order-starved comm#279cf575 passover_picks 26 MED
long-range-dependent migration-rate hurst 2.6417 MED
discord 7799ms profile_vs_mean 1.76974 MED
hot-cpu 10 l2_miss_share 17.4064 MED
order-starved comm#8a5f93b2 passover_picks 21 MED
order-starved comm#8a5f93b2 passover_picks 18 MED
long-range-dependent dispatch-rate hurst 0.845505 MED
recvfrom-slow 250836/pand-ipc duration_ms 1.64073 LOW
recvfrom-slow 250838/pand-ipc duration_ms 1.64008 LOW
recvfrom-slow 250836/pand-ipc duration_ms 1.63836 LOW
recvfrom-slow 250836/pand-ipc duration_ms 1.63668 LOW
recvfrom-slow 250836/pand-ipc duration_ms 1.6335 LOW
ioctl-slow 250854/python3 duration_ms 0.00084 LOW
ioctl-slow 250854/python3 duration_ms 0.00053 LOW
ioctl-slow 250854/python3 duration_ms 0.00051 LOW
ioctl-slow 250854/python3 duration_ms 0.00045 LOW
ioctl-slow 250854/python3 duration_ms 0.00039 LOW
CAPTURE complete -- 1698625 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 218k wake2run; p50 1us p99 25us p999 1537us worst 4805us; 99.5% fast(<100us) / 0.2% mid / 0.3% tick-floor(>=900us); 0.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~88487 distinct values; inversion ratio 0.26
LOCATED: 8 structured region(s) FEW_UNIQUE[0%..2%], FEW_UNIQUE[3%..14%], FEW_UNIQUE[14%..29%], FEW_UNIQUE[29%..40%], FEW_UNIQUE[40%..48%] +3 more
REPORT dispatch-stall
VERDICT: 744 saturated wakes over the 900us floor; PREEMPT-STARVED 76% (0 intervening picks) / ORDER-STARVED 24% (CPU served others first); avg 1.8 pass-overs, p99 4 pass-overs
WORKLOAD ipc-PANDEMONIUM-ADAPTIVE-16c
THERMAL/POWER
cpu temp peak 58.0 C avg 50.4 C
power avg 111.2 W peak 143.0 W
idle C2 avg 56.5% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
order-starved comm#8a5f93b2 passover_picks 106 HIGH
held-cpu comm#94672850 held_ms 53.6494 HIGH
hot-waker 251422 wakes_issued 79826 HIGH
self-preempt tid=251361 stint_wake_ratio 43648 HIGH
self-preempt tid=251351 stint_wake_ratio 41378 HIGH
self-preempt tid=251354 stint_wake_ratio 35638 HIGH
self-preempt tid=251352 stint_wake_ratio 29341 HIGH
order-starved comm#8a5f93b2 passover_picks 57 HIGH
held-cpu comm#c233d180 held_ms 40.179 HIGH
self-preempt tid=251353 stint_wake_ratio 24233 HIGH
self-preempt tid=251347 stint_wake_ratio 23248 HIGH
self-preempt tid=251348 stint_wake_ratio 14485 HIGH
order-starved comm#8a5f93b2 passover_picks 50 HIGH
held-cpu comm#c233d180 held_ms 6.80869 HIGH
self-preempt tid=251350 stint_wake_ratio 13652 HIGH
long-range-dependent dispatch-rate hurst 0.869398 MED
discord 1291ms profile_vs_mean 1.66278 MED
hot-cpu 7 l2_miss_share 16.2954 MED
long-range-dependent migration-rate hurst 0.542905 MED
recvfrom-slow 251397/pand-ipc duration_ms 2.18242 LOW
recvfrom-slow 251395/pand-ipc duration_ms 2.17796 LOW
recvfrom-slow 251397/pand-ipc duration_ms 2.02692 LOW
recvfrom-slow 251395/pand-ipc duration_ms 2.02541 LOW
recvfrom-slow 251397/pand-ipc duration_ms 2.02004 LOW
ioctl-slow 251413/python3 duration_ms 0.00086 LOW
ioctl-slow 251413/python3 duration_ms 0.00055 LOW
ioctl-slow 251413/python3 duration_ms 0.00052 LOW
ioctl-slow 251413/python3 duration_ms 0.00044 LOW
ioctl-slow 251413/python3 duration_ms 0.00038 LOW
CAPTURE complete -- 1854074 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 242k wake2run; p50 3us p99 12us p999 999us worst 2671us; 99.3% fast(<100us) / 0.5% mid / 0.2% tick-floor(>=900us); 0.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~166219 distinct values; inversion ratio 0.48
LOCATED: 7 structured region(s) FEW_UNIQUE[10%..11%], FEW_UNIQUE[11%..11%], FEW_UNIQUE[14%..14%], NEARLY_SORTED[17%..17%], FEW_UNIQUE[23%..24%] +2 more
REPORT dispatch-stall
VERDICT: 451 saturated wakes over the 900us floor; PREEMPT-STARVED 21% (0 intervening picks) / ORDER-STARVED 79% (CPU served others first); avg 1.5 pass-overs, p99 7 pass-overs
WORKLOAD ipc-PANDEMONIUM-BPF-16c
THERMAL/POWER
cpu temp peak 57.2 C avg 55.1 C
power avg 122.5 W peak 142.8 W
idle C2 avg 34.0% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
order-starved comm#8a5f93b2 passover_picks 48 HIGH
held-cpu comm#c233d180 held_ms 20.3104 HIGH
hot-waker 251121 wakes_issued 79922 HIGH
self-preempt tid=251055 stint_wake_ratio 49174 HIGH
self-preempt tid=251051 stint_wake_ratio 37545 HIGH
self-preempt tid=251048 stint_wake_ratio 29974 HIGH
self-preempt tid=251050 stint_wake_ratio 29396 HIGH
order-starved comm#8a5f93b2 passover_picks 30 HIGH
held-cpu comm#94672850 held_ms 17.9817 HIGH
self-preempt tid=251053 stint_wake_ratio 28889 HIGH
self-preempt tid=251064 stint_wake_ratio 24552 HIGH
self-preempt tid=251054 stint_wake_ratio 19391 HIGH
order-starved comm#8a5f93b2 passover_picks 29 HIGH
held-cpu comm#c233d180 held_ms 5.76678 HIGH
self-preempt tid=251056 stint_wake_ratio 11045 HIGH
long-range-dependent dispatch-rate hurst 0.889317 MED
discord 5114ms profile_vs_mean 1.72328 MED
hot-cpu 4 l2_miss_share 14.4953 MED
long-range-dependent migration-rate hurst 0.570152 MED
recvfrom-slow 251093/pand-ipc duration_ms 3.03604 LOW
recvfrom-slow 251095/pand-ipc duration_ms 3.03546 LOW
recvfrom-slow 251095/pand-ipc duration_ms 3.00259 LOW
recvfrom-slow 251093/pand-ipc duration_ms 2.99591 LOW
recvfrom-slow 251095/pand-ipc duration_ms 2.03856 LOW
ioctl-slow 251112/python3 duration_ms 0.00099 LOW
ioctl-slow 251112/python3 duration_ms 0.00054 LOW
ioctl-slow 251112/python3 duration_ms 0.00048 LOW
ioctl-slow 251112/python3 duration_ms 0.00046 LOW
ioctl-slow 251112/python3 duration_ms 0.00039 LOW
CAPTURE complete -- 1865493 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 242k wake2run; p50 3us p99 12us p999 993us worst 2529us; 99.3% fast(<100us) / 0.5% mid / 0.1% tick-floor(>=900us); 0.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~158845 distinct values; inversion ratio 0.49
LOCATED: 12 structured region(s) FEW_UNIQUE[6%..6%], FEW_UNIQUE[6%..7%], FEW_UNIQUE[7%..7%], FEW_UNIQUE[12%..12%], FEW_UNIQUE[17%..17%] +7 more
REPORT dispatch-stall
VERDICT: 351 saturated wakes over the 900us floor; PREEMPT-STARVED 13% (0 intervening picks) / ORDER-STARVED 87% (CPU served others first); avg 1.5 pass-overs, p99 5 pass-overs
WORKLOAD pcpu-burst-16c-EEVDF
THERMAL/POWER
cpu temp peak 61.2 C avg 59.6 C
power avg 134.6 W peak 157.3 W
idle C2 avg 35.8% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
order-starved comm#328ab0c6 passover_picks 11371 HIGH
kthread-strand comm#cfa272d9 max_strand_ms 190 HIGH
doublefree-logic 251658/0x0000557b86ce5b50 bytes 24 HIGH
self-preempt tid=1827 stint_wake_ratio 17675 HIGH
doublefree-logic 251657/0x00005590cc00cb50 bytes 24 HIGH
doublefree-logic 251656/0x00005600f3aedb50 bytes 24 HIGH
self-preempt tid=1826 stint_wake_ratio 17649 HIGH
doublefree-logic 251655/0x000056051ccf6b50 bytes 24 HIGH
doublefree-logic 251654/0x00005563cc9dbb50 bytes 24 HIGH
self-preempt tid=1650 stint_wake_ratio 17082 HIGH
doublefree-logic 251653/0x0000560b3aba8b50 bytes 24 HIGH
doublefree-logic 251650/0x000055bc55aacb50 bytes 24 HIGH
self-preempt tid=1717 stint_wake_ratio 16943 HIGH
doublefree-logic 251652/0x000055e8a3fdcb50 bytes 24 HIGH
order-starved comm#328ab0c6 passover_picks 11233 HIGH
doublefree-logic 251651/0x000055aac5bafb50 bytes 24 HIGH
self-preempt tid=1718 stint_wake_ratio 16882 HIGH
doublefree-logic 251648/0x0000558efe777b50 bytes 24 HIGH
doublefree-logic 251647/0x00005654f6a37b50 bytes 24 HIGH
self-preempt tid=1652 stint_wake_ratio 16744 HIGH
doublefree-logic 251645/0x0000563109d0eb50 bytes 24 HIGH
doublefree-logic 251643/0x0000561fffa48b50 bytes 24 HIGH
self-preempt tid=1649 stint_wake_ratio 16447 HIGH
doublefree-logic 251642/0x0000556c1c77fb50 bytes 24 HIGH
doublefree-logic 251641/0x000055f10c1d8b50 bytes 24 HIGH
order-starved comm#328ab0c6 passover_picks 8042 HIGH
doublefree-logic 251639/0x000055dc2d50bb50 bytes 24 HIGH
self-preempt tid=1651 stint_wake_ratio 16310 HIGH
long-range-dependent dispatch-rate hurst 0.857913 MED
discord 3369ms profile_vs_mean 1.63472 MED
mean-reverting migration-rate hurst -2.27585 MED
poll-slow 251627/pandemonium duration_ms 0.00175 LOW
hot-cpu 15 l2_miss_share 7.04579 LOW
poll-slow 251634/pandemonium duration_ms 0.00143 LOW
poll-slow 251628/pandemonium duration_ms 0.00141 LOW
poll-slow 251629/pandemonium duration_ms 0.00131 LOW
poll-slow 251635/pandemonium duration_ms 0.0013 LOW
CAPTURE complete -- 613699 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 65.9k wake2run; p50 1642us p99 14504us p999 19485us worst 44605us; 31.1% fast(<100us) / 6.5% mid / 62.5% tick-floor(>=900us); 0.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~64850 distinct values; inversion ratio 0.43
REPORT dispatch-stall
VERDICT: 41.2k saturated wakes over the 900us floor; PREEMPT-STARVED 0% (0 intervening picks) / ORDER-STARVED 100% (CPU served others first); avg 8.2 pass-overs, p99 31 pass-overs
REPORT kstrand
VERDICT: 1 strands across 1 per-CPU kthreads; worst HELD strand 190.0ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#cfa272d9 15 1 190.0 190.0 1 0 comm#c233d180 13%
WORKLOAD pcpu-burst-16c-PANDEMONIUM
THERMAL/POWER
cpu temp peak 61.8 C avg 58.7 C
power avg 138.6 W peak 160.0 W
idle C2 avg 27.2% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
order-starved comm#328ab0c6 passover_picks 12732 HIGH
doublefree-logic 253396/0x000055b03f2a8b50 bytes 24 HIGH
self-preempt tid=1652 stint_wake_ratio 18680 HIGH
doublefree-logic 253395/0x000056303967fb50 bytes 24 HIGH
doublefree-logic 253392/0x00005633f34ebb50 bytes 24 HIGH
self-preempt tid=1827 stint_wake_ratio 18508 HIGH
doublefree-logic 253394/0x0000556c6cf0fb50 bytes 24 HIGH
doublefree-logic 253393/0x000055873bc97b50 bytes 24 HIGH
self-preempt tid=1717 stint_wake_ratio 18487 HIGH
doublefree-logic 253391/0x0000556a6dea5b50 bytes 24 HIGH
doublefree-logic 253390/0x00005625ad017b50 bytes 24 HIGH
self-preempt tid=1718 stint_wake_ratio 18481 HIGH
doublefree-logic 253389/0x000055cb2641cb50 bytes 24 HIGH
order-starved comm#328ab0c6 passover_picks 12314 HIGH
doublefree-logic 253388/0x0000556bfa92ab50 bytes 24 HIGH
self-preempt tid=1651 stint_wake_ratio 18446 HIGH
doublefree-logic 253387/0x000055b4345e5b50 bytes 24 HIGH
doublefree-logic 253379/0x000055b32510cb50 bytes 24 HIGH
self-preempt tid=1826 stint_wake_ratio 18420 HIGH
doublefree-logic 253384/0x000055c4738d9b50 bytes 24 HIGH
doublefree-logic 253385/0x000055ef66a10b50 bytes 24 HIGH
self-preempt tid=1649 stint_wake_ratio 18392 HIGH
doublefree-logic 253381/0x000055d89608cb50 bytes 24 HIGH
doublefree-logic 253376/0x000055f05da21b50 bytes 24 HIGH
order-starved comm#328ab0c6 passover_picks 12297 HIGH
doublefree-logic 253375/0x000055c5bf93eb50 bytes 24 HIGH
self-preempt tid=1650 stint_wake_ratio 18368 HIGH
long-range-dependent dispatch-rate hurst 0.87895 MED
discord 16281ms profile_vs_mean 2.23148 MED
poll-slow 253369/pandemonium duration_ms 0.00134 LOW
hot-cpu 7 l2_miss_share 9.10551 LOW
poll-slow 253368/pandemonium duration_ms 0.00133 LOW
poll-slow 253373/pandemonium duration_ms 0.00131 LOW
poll-slow 253370/pandemonium duration_ms 0.00131 LOW
poll-slow 253367/pandemonium duration_ms 0.00131 LOW
CAPTURE complete -- 795415 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 62.1k wake2run; p50 2695us p99 12391us p999 14998us worst 17072us; 22.4% fast(<100us) / 7.5% mid / 70.1% tick-floor(>=900us); 0.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~62148 distinct values; inversion ratio 0.41
LOCATED: 5 structured region(s) NEARLY_SORTED[20%..21%], NEARLY_SORTED[28%..29%], NEARLY_SORTED[37%..38%], NEARLY_SORTED[50%..51%], NEARLY_SORTED[63%..66%]
REPORT dispatch-stall
VERDICT: 43.6k saturated wakes over the 900us floor; PREEMPT-STARVED 0% (0 intervening picks) / ORDER-STARVED 100% (CPU served others first); avg 11.3 pass-overs, p99 46 pass-overs
WORKLOAD pcpu-burst-16c-scx_beerland
THERMAL/POWER
cpu temp peak 62.8 C avg 60.0 C
power avg 142.7 W peak 158.6 W
idle C2 avg 17.8% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
order-starved comm#328ab0c6 passover_picks 12739 HIGH
doublefree-logic 272516/0x0000560c68ac8b50 bytes 24 HIGH
self-preempt tid=1717 stint_wake_ratio 18920 HIGH
doublefree-logic 272515/0x000056536ad3cb50 bytes 24 HIGH
doublefree-logic 272514/0x0000556888098b50 bytes 24 HIGH
self-preempt tid=1649 stint_wake_ratio 18920 HIGH
doublefree-logic 272513/0x00005616634fbb50 bytes 24 HIGH
doublefree-logic 272512/0x000055aa977e0b50 bytes 24 HIGH
self-preempt tid=1827 stint_wake_ratio 18917 HIGH
doublefree-logic 272511/0x000056463ea9cb50 bytes 24 HIGH
doublefree-logic 272510/0x000055a98fb32b50 bytes 24 HIGH
self-preempt tid=1651 stint_wake_ratio 18916 HIGH
doublefree-logic 272509/0x000055ce1080fb50 bytes 24 HIGH
order-starved comm#328ab0c6 passover_picks 11970 HIGH
doublefree-logic 272508/0x000055a31bfd2b50 bytes 24 HIGH
self-preempt tid=1652 stint_wake_ratio 18915 HIGH
doublefree-logic 272507/0x000056098dbc4b50 bytes 24 HIGH
doublefree-logic 272506/0x0000556d47750b50 bytes 24 HIGH
self-preempt tid=1718 stint_wake_ratio 18914 HIGH
doublefree-logic 272505/0x0000559f2f3f5b50 bytes 24 HIGH
doublefree-logic 272502/0x0000565558f5cb50 bytes 24 HIGH
self-preempt tid=1826 stint_wake_ratio 18910 HIGH
doublefree-logic 272500/0x000055ea9c1cfb50 bytes 24 HIGH
doublefree-logic 272498/0x000055b039de5b50 bytes 24 HIGH
order-starved comm#328ab0c6 passover_picks 11941 HIGH
doublefree-logic 272497/0x000055f29e510b50 bytes 24 HIGH
self-preempt tid=1650 stint_wake_ratio 18904 HIGH
kthread-strand comm#cfa272d9 max_strand_ms 22.9992 MED
long-range-dependent dispatch-rate hurst 0.811871 MED
discord 15297ms profile_vs_mean 1.86676 MED
kthread-strand comm#2f099b5a max_strand_ms 13.9988 MED
kthread-strand comm#ff850676 max_strand_ms 11.0016 MED
kthread-strand comm#eb6af029 max_strand_ms 8.13234 MED
kthread-strand comm#70339bdc max_strand_ms 7.99982 MED
kthread-strand comm#1beaeefc max_strand_ms 7.84091 MED
kthread-strand comm#dca24885 max_strand_ms 6.06195 MED
mean-reverting migration-rate hurst -2.2133 MED
poll-slow 272491/pandemonium duration_ms 0.00161 LOW
hot-cpu 7 l2_miss_share 6.71137 LOW
poll-slow 272485/pandemonium duration_ms 0.00151 LOW
poll-slow 272489/pandemonium duration_ms 0.00128 LOW
poll-slow 272486/pandemonium duration_ms 0.00125 LOW
poll-slow 272492/pandemonium duration_ms 0.00124 LOW
CAPTURE complete -- 524278 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 41.6k wake2run; p50 4231us p99 15997us p999 18005us worst 39001us; 21.7% fast(<100us) / 3.0% mid / 75.2% tick-floor(>=900us); 0.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~41619 distinct values; inversion ratio 0.41
REPORT dispatch-stall
VERDICT: 31.3k saturated wakes over the 900us floor; PREEMPT-STARVED 0% (0 intervening picks) / ORDER-STARVED 100% (CPU served others first); avg 9.6 pass-overs, p99 25 pass-overs
REPORT kstrand
VERDICT: 7 strands across 7 per-CPU kthreads; worst HELD strand 23.0ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#cfa272d9 15 1 23.0 23.0 1 0 comm#7e44bede 52%
comm#2f099b5a 2 1 14.0 14.0 1 0 comm#8a5f93b2 14%
comm#ff850676 2 1 11.0 11.0 1 0 comm#f7d5fbc2 55%
comm#eb6af029 15 1 8.1 8.1 1 0 comm#c233d180 49%
comm#70339bdc 4 1 8.0 8.0 1 0 comm#8a5f93b2 25%
comm#1beaeefc 4 1 7.8 7.8 1 0 comm#7e44bede 99%
comm#dca24885 9 1 6.1 6.1 1 0 comm#7e44bede 100%
WORKLOAD pcpu-burst-16c-scx_bpfland
THERMAL/POWER
cpu temp peak 62.8 C avg 59.6 C
power avg 140.5 W peak 159.4 W
idle C2 avg 20.9% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
order-starved comm#c233d180 passover_picks 3437 HIGH
doublefree-logic 255756/0x00005609b842a680 bytes 240 HIGH
self-preempt tid=1652 stint_wake_ratio 13696 HIGH
doublefree-logic 255773/0x0000556aa6c89b50 bytes 24 HIGH
doublefree-logic 255772/0x000055e10f73ab50 bytes 24 HIGH
self-preempt tid=1827 stint_wake_ratio 13696 HIGH
doublefree-logic 255770/0x00005629412fbb50 bytes 24 HIGH
doublefree-logic 255771/0x00005609b842cb50 bytes 24 HIGH
self-preempt tid=1717 stint_wake_ratio 13694 HIGH
doublefree-logic 255769/0x00005584e66e1b50 bytes 24 HIGH
doublefree-logic 255768/0x000055a6b08d3b50 bytes 24 HIGH
self-preempt tid=1651 stint_wake_ratio 13694 HIGH
doublefree-logic 255767/0x0000556d65a9cb50 bytes 24 HIGH
order-starved comm#5a4f434f passover_picks 2892 HIGH
doublefree-logic 255766/0x00005588d365ab50 bytes 24 HIGH
doublefree-logic 255765/0x000055b1c4549b50 bytes 24 HIGH
self-preempt tid=1718 stint_wake_ratio 13693 HIGH
doublefree-logic 255764/0x0000556c876ecb50 bytes 24 HIGH
doublefree-logic 255763/0x000055854cdb2b50 bytes 24 HIGH
self-preempt tid=1649 stint_wake_ratio 13690 HIGH
doublefree-logic 255762/0x000055a81df51b50 bytes 24 HIGH
doublefree-logic 255759/0x00005649b9440b50 bytes 24 HIGH
self-preempt tid=1826 stint_wake_ratio 13687 HIGH
doublefree-logic 255757/0x0000559348084b50 bytes 24 HIGH
doublefree-logic 255755/0x0000558325f4ab50 bytes 24 HIGH
order-starved comm#c233d180 passover_picks 1389 HIGH
doublefree-logic 255754/0x0000564d5ceb4b50 bytes 24 HIGH
self-preempt tid=1650 stint_wake_ratio 13685 HIGH
kthread-strand comm#cfa272d9 max_strand_ms 59.9996 MED
long-range-dependent dispatch-rate hurst 0.81703 MED
discord 1382ms profile_vs_mean 2.86655 MED
kthread-strand comm#eb6af029 max_strand_ms 52.3478 MED
kthread-strand comm#ff850676 max_strand_ms 31.9995 MED
kthread-strand comm#288f3ed3 max_strand_ms 20.9845 MED
kthread-strand comm#288f4086 max_strand_ms 17.9995 MED
kthread-strand comm#288f4fd1 max_strand_ms 15.9945 MED
kthread-strand comm#288f3d20 max_strand_ms 11.995 MED
kthread-strand comm#288f43ec max_strand_ms 11.9773 MED
kthread-strand comm#eb6af6f5 max_strand_ms 10.9992 MED
kthread-strand comm#288f4752 max_strand_ms 10.9893 MED
kthread-strand comm#288f459f max_strand_ms 10.0015 MED
kthread-strand comm#1beaf0af max_strand_ms 9.00015 MED
kthread-strand comm#ca8154b4 max_strand_ms 8.00137 MED
kthread-strand comm#12f62659 max_strand_ms 7.0029 MED
kthread-strand comm#493ffe92 max_strand_ms 7.0004 MED
kthread-strand comm#e3146c3c max_strand_ms 7.00011 MED
kthread-strand comm#eb6aee76 max_strand_ms 6.99113 MED
kthread-strand comm#fa4c2883 max_strand_ms 6.00008 MED
kthread-strand comm#eb6af38f max_strand_ms 5.9989 MED
kthread-strand comm#eb6af1dc max_strand_ms 5.99838 MED
kthread-strand comm#288f4239 max_strand_ms 5.99815 MED
kthread-strand comm#eb6af542 max_strand_ms 5.04844 MED
kthread-strand comm#0dbd5298 max_strand_ms 5.0076 MED
kthread-strand comm#d4e6d4f1 max_strand_ms 5.00318 MED
kthread-strand comm#da1fa6ff max_strand_ms 5.00044 MED
mean-reverting migration-rate hurst -2.71505 MED
poll-slow 255749/pandemonium duration_ms 0.00167 LOW
hot-cpu 2 l2_miss_share 6.67264 LOW
poll-slow 255742/pandemonium duration_ms 0.00167 LOW
poll-slow 255747/pandemonium duration_ms 0.00131 LOW
poll-slow 255750/pandemonium duration_ms 0.0013 LOW
poll-slow 255748/pandemonium duration_ms 0.00127 LOW
CAPTURE complete -- 476441 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 21.9k wake2run; p50 998us p99 115994us p999 140005us worst 168000us; 43.4% fast(<100us) / 2.5% mid / 54.0% tick-floor(>=900us); 0.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~21895 distinct values; inversion ratio 0.45
LOCATED: 2 structured region(s) NEARLY_SORTED[54%..56%], NEARLY_SORTED[80%..82%]
REPORT dispatch-stall
VERDICT: 11.8k saturated wakes over the 900us floor; PREEMPT-STARVED 0% (0 intervening picks) / ORDER-STARVED 100% (CPU served others first); avg 26.2 pass-overs, p99 211 pass-overs
REPORT kstrand
VERDICT: 68 strands across 25 per-CPU kthreads; worst HELD strand 60.0ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#cfa272d9 15 8 60.0 60.0 8 0 comm#c233d180 51%
comm#eb6af029 15 20 52.3 52.3 20 0 comm#c233d180 57%
comm#ff850676 2 8 32.0 32.0 8 0 comm#f7d5fbc2 52%
comm#288f3ed3 2 3 21.0 21.0 3 0 comm#8a5f93b2 43%
comm#288f4086 1 1 18.0 18.0 1 0 comm#8a5f93b2 28%
comm#288f4fd1 8 1 16.0 16.0 1 0 comm#7e44bede 25%
comm#288f3d20 3 1 12.0 12.0 1 0 comm#7e44bede 40%
comm#288f43ec 7 1 12.0 12.0 1 0 comm#8a5f93b2 33%
comm#eb6af6f5 11 3 11.0 11.0 3 0 comm#8a5f93b2 45%
comm#288f4752 5 1 11.0 11.0 1 0 comm#8a5f93b2 62%
comm#288f459f 6 2 10.0 10.0 2 0 comm#7e44bede 52%
comm#1beaf0af 4 2 9.0 9.0 2 0 comm#7e44bede 87%
comm#ca8154b4 14 2 8.0 8.0 2 0 comm#7e44bede 49%
comm#12f62659 6 1 7.0 7.0 1 0 comm#7e44bede 43%
comm#493ffe92 10 1 7.0 7.0 1 0 comm#7e44bede 57%
comm#e3146c3c 11 1 7.0 7.0 1 0 comm#7e44bede 70%
comm#eb6aee76 14 1 7.0 7.0 1 0 comm#7e44bede 57%
comm#fa4c2883 3 1 6.0 6.0 1 0 comm#94672850 50%
comm#eb6af38f 13 2 6.0 6.0 2 0 comm#7e44bede 65%
comm#eb6af1dc 12 2 6.0 6.0 2 0 comm#7e44bede 50%
WORKLOAD pcpu-burst-16c-scx_cake
THERMAL/POWER
cpu temp peak 62.8 C avg 58.9 C
power avg 135.0 W peak 154.1 W
idle C2 avg 22.8% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
order-starved comm#5a4f434f passover_picks 3144 HIGH
doublefree-logic 269357/0x000055754ce5ab50 bytes 24 HIGH
self-preempt tid=1652 stint_wake_ratio 22002 HIGH
doublefree-logic 269358/0x0000560e0b1b2b50 bytes 24 HIGH
doublefree-logic 269356/0x000055d6b424db50 bytes 24 HIGH
self-preempt tid=1650 stint_wake_ratio 21879 HIGH
doublefree-logic 269355/0x000055d36f273b50 bytes 24 HIGH
doublefree-logic 269354/0x0000562a4ab4fb50 bytes 24 HIGH
self-preempt tid=1827 stint_wake_ratio 17605 HIGH
doublefree-logic 269353/0x000055e797b16b50 bytes 24 HIGH
doublefree-logic 269352/0x0000564c9737db50 bytes 24 HIGH
self-preempt tid=1718 stint_wake_ratio 17512 HIGH
doublefree-logic 269351/0x000055ed35cddb50 bytes 24 HIGH
order-starved comm#94672850 passover_picks 2848 HIGH
doublefree-logic 269350/0x000055cbf8c6eb50 bytes 24 HIGH
self-preempt tid=1649 stint_wake_ratio 17273 HIGH
doublefree-logic 269349/0x000055f069c8eb50 bytes 24 HIGH
doublefree-logic 269348/0x0000555fbea32b50 bytes 24 HIGH
self-preempt tid=1717 stint_wake_ratio 17248 HIGH
doublefree-logic 269347/0x0000557731ab5b50 bytes 24 HIGH
doublefree-logic 269346/0x0000562e26bd3b50 bytes 24 HIGH
self-preempt tid=1651 stint_wake_ratio 12687 HIGH
doublefree-logic 269344/0x000055cb59fb9b50 bytes 24 HIGH
doublefree-logic 269343/0x0000555f5b79db50 bytes 24 HIGH
order-starved comm#c233d180 passover_picks 1180 HIGH
doublefree-logic 269341/0x000055ed8ed57b50 bytes 24 HIGH
self-preempt tid=1826 stint_wake_ratio 12564 HIGH
kthread-strand comm#4f7aa3e7 max_strand_ms 29.1276 MED
long-range-dependent dispatch-rate hurst 0.780515 MED
hot-cpu 2 l2_miss_share 13.7131 MED
kthread-strand comm#ca8154b4 max_strand_ms 26.9987 MED
kthread-strand comm#fa4c2883 max_strand_ms 23.9997 MED
kthread-strand comm#180cd039 max_strand_ms 23.9996 MED
kthread-strand comm#d4e6d4f1 max_strand_ms 21.9991 MED
kthread-strand comm#658ebbd8 max_strand_ms 21.998 MED
kthread-strand comm#da1fa8b2 max_strand_ms 21.9967 MED
kthread-strand comm#12f62659 max_strand_ms 21.993 MED
kthread-strand comm#e3146c3c max_strand_ms 20.9949 MED
kthread-strand comm#493ffe92 max_strand_ms 19.9992 MED
kthread-strand comm#0357d927 max_strand_ms 19.9945 MED
kthread-strand comm#288f4239 max_strand_ms 19.988 MED
kthread-strand comm#eb6af542 max_strand_ms 19.7472 MED
kthread-strand comm#e1c397f1 max_strand_ms 19.0015 MED
kthread-strand comm#1beaeefc max_strand_ms 19.0001 MED
kthread-strand comm#0dbd5298 max_strand_ms 17.999 MED
kthread-strand comm#dca24885 max_strand_ms 16.9944 MED
kthread-strand comm#ff850676 max_strand_ms 14.0071 MED
kthread-strand comm#cfa26f73 max_strand_ms 14.0008 MED
kthread-strand comm#288f459f max_strand_ms 11.9946 MED
kthread-strand comm#2f099b5a max_strand_ms 9.00139 MED
mean-reverting migration-rate hurst -2.2438 MED
poll-slow 269333/pandemonium duration_ms 0.00168 LOW
poll-slow 269335/pandemonium duration_ms 0.0014 LOW
poll-slow 269336/pandemonium duration_ms 0.00136 LOW
poll-slow 269327/pandemonium duration_ms 0.00136 LOW
poll-slow 269330/pandemonium duration_ms 0.00127 LOW
CAPTURE complete -- 545530 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 102k wake2run; p50 18us p99 25001us p999 33006us worst 72000us; 57.2% fast(<100us) / 2.6% mid / 40.1% tick-floor(>=900us); 0.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~98757 distinct values; inversion ratio 0.34
REPORT dispatch-stall
VERDICT: 40.9k saturated wakes over the 900us floor; PREEMPT-STARVED 0% (0 intervening picks) / ORDER-STARVED 100% (CPU served others first); avg 3.0 pass-overs, p99 14 pass-overs
REPORT kstrand
VERDICT: 186 strands across 21 per-CPU kthreads; worst HELD strand 29.1ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#4f7aa3e7 3 2 29.1 29.1 2 0 comm#8a5f93b2 77%
comm#ca8154b4 14 4 27.0 27.0 4 0 comm#8a5f93b2 64%
comm#fa4c2883 3 10 24.0 24.0 10 0 comm#8a5f93b2 50%
comm#180cd039 5 23 24.0 24.0 23 0 comm#8a5f93b2 88%
comm#d4e6d4f1 12 13 22.0 22.0 13 0 comm#8a5f93b2 95%
comm#658ebbd8 0 7 22.0 22.0 7 0 comm#8a5f93b2 95%
comm#da1fa8b2 13 12 22.0 22.0 12 0 comm#8a5f93b2 95%
comm#12f62659 6 16 22.0 22.0 16 0 comm#8a5f93b2 95%
comm#e3146c3c 11 8 21.0 21.0 8 0 comm#8a5f93b2 100%
comm#493ffe92 10 12 20.0 20.0 12 0 comm#8a5f93b2 100%
comm#0357d927 1 18 20.0 20.0 18 0 comm#8a5f93b2 100%
comm#288f4239 0 2 20.0 20.0 2 0 comm#8a5f93b2 100%
comm#eb6af542 10 1 19.7 19.7 1 0 comm#8a5f93b2 95%
comm#e1c397f1 8 7 19.0 19.0 7 0 comm#8a5f93b2 100%
comm#1beaeefc 4 15 19.0 19.0 15 0 comm#8a5f93b2 100%
comm#0dbd5298 7 6 18.0 18.0 6 0 comm#8a5f93b2 100%
comm#dca24885 9 7 17.0 17.0 7 0 comm#8a5f93b2 100%
comm#ff850676 2 18 14.0 14.0 18 0 comm#8a5f93b2 36%
comm#cfa26f73 15 2 14.0 14.0 2 0 comm#c233d180 100%
comm#288f459f 6 1 12.0 12.0 1 0 comm#8a5f93b2 100%
WORKLOAD pcpu-burst-16c-scx_cosmos
THERMAL/POWER
cpu temp peak 63.2 C avg 59.0 C
power avg 138.0 W peak 162.4 W
idle C2 avg 32.2% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
order-starved comm#c233d180 passover_picks 3164 HIGH
doublefree-logic 266322/0x00005645c8576680 bytes 240 HIGH
self-preempt tid=1827 stint_wake_ratio 13970 HIGH
doublefree-logic 266337/0x0000563f98d39b50 bytes 24 HIGH
doublefree-logic 266335/0x0000560653a1cb50 bytes 24 HIGH
self-preempt tid=1650 stint_wake_ratio 13967 HIGH
doublefree-logic 266334/0x000055ff4ae21b50 bytes 24 HIGH
doublefree-logic 266333/0x00005645c8578b50 bytes 24 HIGH
self-preempt tid=1826 stint_wake_ratio 13966 HIGH
doublefree-logic 266332/0x00005621c11eeb50 bytes 24 HIGH
doublefree-logic 266331/0x0000563a3eebbb50 bytes 24 HIGH
self-preempt tid=1649 stint_wake_ratio 13966 HIGH
doublefree-logic 266330/0x000055ba6e90fb50 bytes 24 HIGH
order-starved comm#5a4f434f passover_picks 3051 HIGH
doublefree-logic 266329/0x0000556d50d05b50 bytes 24 HIGH
doublefree-logic 266328/0x00005557a758db50 bytes 24 HIGH
self-preempt tid=1651 stint_wake_ratio 13966 HIGH
doublefree-logic 266327/0x000055e157b92b50 bytes 24 HIGH
doublefree-logic 266326/0x0000560760bb2b50 bytes 24 HIGH
self-preempt tid=1717 stint_wake_ratio 13965 HIGH
doublefree-logic 266325/0x000055bafc225b50 bytes 24 HIGH
doublefree-logic 266324/0x0000562e152acb50 bytes 24 HIGH
self-preempt tid=1652 stint_wake_ratio 13965 HIGH
doublefree-logic 266323/0x000055d005f7ab50 bytes 24 HIGH
doublefree-logic 266320/0x000055eb1a26ab50 bytes 24 HIGH
order-starved comm#328ab0c6 passover_picks 1636 HIGH
doublefree-logic 266318/0x000056026a634b50 bytes 24 HIGH
self-preempt tid=1718 stint_wake_ratio 13962 HIGH
kthread-strand comm#eb6af029 max_strand_ms 62.9957 MED
long-range-dependent dispatch-rate hurst 0.825526 MED
discord 19017ms profile_vs_mean 2.67519 MED
kthread-strand comm#cfa272d9 max_strand_ms 62.083 MED
kthread-strand comm#cfa26f73 max_strand_ms 58.9956 MED
kthread-strand comm#fa4c2883 max_strand_ms 39.0021 MED
kthread-strand comm#ff850676 max_strand_ms 34.0087 MED
kthread-strand comm#180cd039 max_strand_ms 21.014 MED
kthread-strand comm#ca8154b4 max_strand_ms 12.9991 MED
kthread-strand comm#658ebbd8 max_strand_ms 10.001 MED
kthread-strand comm#288f4239 max_strand_ms 8.00475 MED
mean-reverting migration-rate hurst -2.1452 MED
poll-slow 266304/pandemonium duration_ms 0.0017 LOW
hot-cpu 4 l2_miss_share 6.45365 LOW
poll-slow 266310/pandemonium duration_ms 0.00146 LOW
poll-slow 266306/pandemonium duration_ms 0.00144 LOW
poll-slow 266307/pandemonium duration_ms 0.00142 LOW
poll-slow 266308/pandemonium duration_ms 0.0014 LOW
CAPTURE complete -- 389844 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 19.6k wake2run; p50 1998us p99 98894us p999 131003us worst 149013us; 44.5% fast(<100us) / 2.4% mid / 53.2% tick-floor(>=900us); 0.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~19614 distinct values; inversion ratio 0.39
LOCATED: 2 structured region(s) NEARLY_SORTED[60%..63%], NEARLY_SORTED[84%..86%]
REPORT dispatch-stall
VERDICT: 10.4k saturated wakes over the 900us floor; PREEMPT-STARVED 0% (0 intervening picks) / ORDER-STARVED 100% (CPU served others first); avg 21.6 pass-overs, p99 136 pass-overs
REPORT kstrand
VERDICT: 85 strands across 9 per-CPU kthreads; worst HELD strand 63.0ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#eb6af029 15 8 63.0 63.0 8 0 comm#c233d180 75%
comm#cfa272d9 15 50 62.1 62.1 50 0 comm#c233d180 67%
comm#cfa26f73 15 6 59.0 59.0 6 0 comm#c233d180 83%
comm#fa4c2883 3 4 39.0 39.0 4 0 comm#7e44bede 100%
comm#ff850676 2 8 34.0 34.0 8 0 comm#7e44bede 99%
comm#180cd039 5 1 21.0 21.0 1 0 comm#7e44bede 99%
comm#ca8154b4 14 2 13.0 13.0 2 0 comm#7e44bede 100%
comm#658ebbd8 0 5 10.0 10.0 5 0 comm#7e44bede 79%
comm#288f4239 0 1 8.0 8.0 1 0 comm#7e44bede 100%
WORKLOAD pcpu-burst-16c-scx_flash
THERMAL/POWER
cpu temp peak 63.2 C avg 60.1 C
power avg 148.8 W peak 159.8 W
idle C2 avg 11.1% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
order-starved comm#5a4f434f passover_picks 3191 HIGH
kthread-strand comm#cfa26f73 max_strand_ms 106.002 HIGH
doublefree-logic 270452/0x0000557ec28dcb50 bytes 24 HIGH
self-preempt tid=1650 stint_wake_ratio 13549 HIGH
doublefree-logic 270451/0x0000556a1c892b50 bytes 24 HIGH
doublefree-logic 270450/0x0000556ed7f6db50 bytes 24 HIGH
self-preempt tid=1651 stint_wake_ratio 13537 HIGH
doublefree-logic 270449/0x00005556e8d88b50 bytes 24 HIGH
doublefree-logic 270448/0x0000555e8dd45b50 bytes 24 HIGH
self-preempt tid=1717 stint_wake_ratio 13536 HIGH
doublefree-logic 270447/0x0000560e0e18db50 bytes 24 HIGH
doublefree-logic 270446/0x000055b9aad7cb50 bytes 24 HIGH
self-preempt tid=1649 stint_wake_ratio 13533 HIGH
doublefree-logic 270445/0x0000557d64250b50 bytes 24 HIGH
order-starved comm#c233d180 passover_picks 2714 HIGH
doublefree-logic 270444/0x00005555c8117b50 bytes 24 HIGH
self-preempt tid=1718 stint_wake_ratio 13533 HIGH
doublefree-logic 270443/0x000055b182312b50 bytes 24 HIGH
doublefree-logic 270442/0x0000558420af6b50 bytes 24 HIGH
self-preempt tid=1827 stint_wake_ratio 13529 HIGH
doublefree-logic 270441/0x000055f311668b50 bytes 24 HIGH
doublefree-logic 270440/0x00005562c968cb50 bytes 24 HIGH
self-preempt tid=1652 stint_wake_ratio 13528 HIGH
doublefree-logic 270438/0x000055fe6fa54b50 bytes 24 HIGH
doublefree-logic 270436/0x0000558c4dcdbb50 bytes 24 HIGH
order-starved comm#328ab0c6 passover_picks 1726 HIGH
doublefree-logic 270434/0x00005559d800cb50 bytes 24 HIGH
self-preempt tid=1826 stint_wake_ratio 13526 HIGH
long-range-dependent dispatch-rate hurst 0.769883 MED
discord 5350ms profile_vs_mean 2.64305 MED
kthread-strand comm#ff850676 max_strand_ms 57.3387 MED
kthread-strand comm#0357d927 max_strand_ms 19.158 MED
kthread-strand comm#eb6af029 max_strand_ms 14.8421 MED
kthread-strand comm#288f3ed3 max_strand_ms 5.07562 MED
mean-reverting migration-rate hurst -2.3027 MED
poll-slow 270421/pandemonium duration_ms 0.0016 LOW
hot-cpu 3 l2_miss_share 6.89226 LOW
poll-slow 270427/pandemonium duration_ms 0.00151 LOW
poll-slow 270423/pandemonium duration_ms 0.00143 LOW
poll-slow 270424/pandemonium duration_ms 0.00141 LOW
poll-slow 270422/pandemonium duration_ms 0.00139 LOW
CAPTURE complete -- 524599 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 44.2k wake2run; p50 4003us p99 12001us p999 16313us worst 167000us; 20.8% fast(<100us) / 0.9% mid / 78.3% tick-floor(>=900us); 0.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~44179 distinct values; inversion ratio 0.34
LOCATED: 5 structured region(s) NEARLY_SORTED[32%..35%], NEARLY_SORTED[45%..48%], NEARLY_SORTED[59%..61%], NEARLY_SORTED[73%..75%], NEARLY_SORTED[88%..89%]
REPORT dispatch-stall
VERDICT: 34.6k saturated wakes over the 900us floor; PREEMPT-STARVED 0% (0 intervening picks) / ORDER-STARVED 100% (CPU served others first); avg 9.1 pass-overs, p99 30 pass-overs
REPORT kstrand
VERDICT: 19 strands across 5 per-CPU kthreads; worst HELD strand 106.0ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#cfa26f73 15 9 106.0 106.0 9 0 comm#7e44bede 77%
comm#ff850676 2 4 57.3 57.3 4 0 comm#f7d5fbc2 100%
comm#0357d927 1 1 19.2 19.2 1 0 comm#94672850 100%
comm#eb6af029 15 4 14.8 14.8 4 0 comm#c233d180 70%
comm#288f3ed3 2 1 5.1 5.1 1 0 comm#94672850 98%
WORKLOAD pcpu-burst-16c-scx_flow
THERMAL/POWER
cpu temp peak 60.5 C avg 58.5 C
power avg 137.6 W peak 150.3 W
idle C2 avg 15.6% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
order-starved comm#94672850 passover_picks 3321 HIGH
doublefree-logic 261289/0x000055775980a680 bytes 240 HIGH
self-preempt tid=1650 stint_wake_ratio 24988 HIGH
doublefree-logic 261307/0x000055cd62267b50 bytes 24 HIGH
doublefree-logic 261306/0x000055775980cb50 bytes 24 HIGH
self-preempt tid=1827 stint_wake_ratio 24970 HIGH
doublefree-logic 261305/0x00005558976e8b50 bytes 24 HIGH
doublefree-logic 261304/0x0000557a6084ab50 bytes 24 HIGH
self-preempt tid=1717 stint_wake_ratio 24947 HIGH
doublefree-logic 261303/0x00005646d4eb4b50 bytes 24 HIGH
doublefree-logic 261302/0x000055bb683a8b50 bytes 24 HIGH
self-preempt tid=1651 stint_wake_ratio 24947 HIGH
doublefree-logic 261301/0x0000564088781b50 bytes 24 HIGH
order-starved comm#5a4f434f passover_picks 3287 HIGH
doublefree-logic 261300/0x000055b7db4dcb50 bytes 24 HIGH
doublefree-logic 261299/0x000055566eb16b50 bytes 24 HIGH
self-preempt tid=1718 stint_wake_ratio 24945 HIGH
doublefree-logic 261298/0x0000562e71ec2b50 bytes 24 HIGH
doublefree-logic 261297/0x000055f6b363cb50 bytes 24 HIGH
self-preempt tid=1649 stint_wake_ratio 24944 HIGH
doublefree-logic 261296/0x000055e6e00b3b50 bytes 24 HIGH
doublefree-logic 261295/0x00005650dfa50b50 bytes 24 HIGH
self-preempt tid=1826 stint_wake_ratio 24943 HIGH
doublefree-logic 261294/0x0000556133c75b50 bytes 24 HIGH
doublefree-logic 261290/0x000056331906cb50 bytes 24 HIGH
order-starved comm#328ab0c6 passover_picks 2600 HIGH
doublefree-logic 261288/0x000055931a56bb50 bytes 24 HIGH
self-preempt tid=1652 stint_wake_ratio 23051 HIGH
long-range-dependent dispatch-rate hurst 0.626032 MED
discord 23872ms profile_vs_mean 1.63097 MED
hot-cpu 9 l2_miss_share 17.9423 MED
mean-reverting migration-rate hurst -2.27511 MED
poll-slow 261278/pandemonium duration_ms 0.00178 LOW
poll-slow 261281/pandemonium duration_ms 0.00149 LOW
poll-slow 261279/pandemonium duration_ms 0.00144 LOW
poll-slow 261276/pandemonium duration_ms 0.00134 LOW
poll-slow 261285/pandemonium duration_ms 0.0013 LOW
CAPTURE complete -- 1055040 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 265k wake2run; p50 824us p99 1008us p999 1887us worst 5727us; 26.6% fast(<100us) / 25.5% mid / 47.9% tick-floor(>=900us); 0.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~260755 distinct values; inversion ratio 0.50
LOCATED: 1 structured region(s) NEARLY_SORTED[51%..51%]
REPORT dispatch-stall
VERDICT: 127k saturated wakes over the 900us floor; PREEMPT-STARVED 0% (0 intervening picks) / ORDER-STARVED 100% (CPU served others first); avg 1.1 pass-overs, p99 3 pass-overs
WORKLOAD pcpu-burst-16c-scx_lavd
THERMAL/POWER
cpu temp peak 62.8 C avg 60.2 C
power avg 132.4 W peak 158.8 W
idle C2 avg 32.3% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
order-starved comm#5a4f434f passover_picks 3057 HIGH
kthread-strand comm#ff850676 max_strand_ms 197.009 HIGH
doublefree-logic 259884/0x0000559316133b50 bytes 24 HIGH
self-preempt tid=1826 stint_wake_ratio 11416 HIGH
doublefree-logic 259881/0x000055dd50fbab50 bytes 24 HIGH
doublefree-logic 259878/0x0000558f7f366b50 bytes 24 HIGH
self-preempt tid=1650 stint_wake_ratio 11361 HIGH
doublefree-logic 259877/0x000055f53ad08b50 bytes 24 HIGH
doublefree-logic 259880/0x000055f5fef89b50 bytes 24 HIGH
self-preempt tid=1652 stint_wake_ratio 11328 HIGH
doublefree-logic 259879/0x0000555dd7342b50 bytes 24 HIGH
doublefree-logic 259882/0x000055b811588b50 bytes 24 HIGH
self-preempt tid=1649 stint_wake_ratio 11324 HIGH
doublefree-logic 259876/0x0000557f407cfb50 bytes 24 HIGH
order-starved comm#328ab0c6 passover_picks 1455 HIGH
doublefree-logic 259868/0x000056156b73eb50 bytes 24 HIGH
self-preempt tid=1651 stint_wake_ratio 11292 HIGH
doublefree-logic 259867/0x0000562a07a2db50 bytes 24 HIGH
doublefree-logic 259866/0x000055b7667bcb50 bytes 24 HIGH
self-preempt tid=1827 stint_wake_ratio 11286 HIGH
doublefree-logic 259862/0x0000555a46d94b50 bytes 24 HIGH
doublefree-logic 259861/0x000055a6af08eb50 bytes 24 HIGH
self-preempt tid=1718 stint_wake_ratio 11242 HIGH
doublefree-logic 259860/0x000055a689e3ab50 bytes 24 HIGH
doublefree-logic 259858/0x000056050947fb50 bytes 24 HIGH
order-starved comm#c233d180 passover_picks 1430 HIGH
doublefree-logic 259859/0x000055b692843b50 bytes 24 HIGH
self-preempt tid=1717 stint_wake_ratio 11240 HIGH
long-range-dependent dispatch-rate hurst 0.703738 MED
discord 17584ms profile_vs_mean 2.23225 MED
kthread-strand comm#0878f8ff max_strand_ms 14.9977 MED
kthread-strand comm#0357d927 max_strand_ms 14.9932 MED
kthread-strand comm#0dbd5298 max_strand_ms 13.998 MED
kthread-strand comm#1beaf0af max_strand_ms 13.0008 MED
kthread-strand comm#fa4c2883 max_strand_ms 12.0008 MED
kthread-strand comm#da1fa6ff max_strand_ms 10.9973 MED
kthread-strand comm#180cd039 max_strand_ms 10.0021 MED
kthread-strand comm#12f62659 max_strand_ms 8.99787 MED
kthread-strand comm#e1c397f1 max_strand_ms 8.98987 MED
kthread-strand comm#cfa272d9 max_strand_ms 8.00197 MED
kthread-strand comm#0357d774 max_strand_ms 7.99966 MED
kthread-strand comm#d4e6d4f1 max_strand_ms 7.00028 MED
kthread-strand comm#eb6af029 max_strand_ms 6.09405 MED
kthread-strand comm#eb6af1dc max_strand_ms 6.00572 MED
kthread-strand comm#dca24885 max_strand_ms 5.99603 MED
kthread-strand comm#288f4905 max_strand_ms 5.00632 MED
mean-reverting migration-rate hurst -2.27311 MED
poll-slow 259870/pandemonium duration_ms 0.00161 LOW
hot-cpu 3 l2_miss_share 7.86893 LOW
poll-slow 259872/pandemonium duration_ms 0.00128 LOW
poll-slow 259871/pandemonium duration_ms 0.00127 LOW
poll-slow 259853/pandemonium duration_ms 0.00126 LOW
poll-slow 259873/pandemonium duration_ms 0.00125 LOW
CAPTURE complete -- 311576 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 18.1k wake2run; p50 5996us p99 50003us p999 65003us worst 69007us; 33.1% fast(<100us) / 1.3% mid / 65.6% tick-floor(>=900us); 0.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~18093 distinct values; inversion ratio 0.32
REPORT dispatch-stall
VERDICT: 11.9k saturated wakes over the 900us floor; PREEMPT-STARVED 0% (0 intervening picks) / ORDER-STARVED 100% (CPU served others first); avg 13.4 pass-overs, p99 70 pass-overs
REPORT kstrand
VERDICT: 105 strands across 17 per-CPU kthreads; worst HELD strand 197.0ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#ff850676 2 14 197.0 197.0 14 0 comm#7e44bede 97%
comm#0878f8ff 0 22 15.0 15.0 22 0 comm#7e44bede 33%
comm#0357d927 1 5 15.0 15.0 5 0 comm#8a5f93b2 67%
comm#0dbd5298 7 10 14.0 14.0 10 0 comm#7e44bede 43%
comm#1beaf0af 4 9 13.0 13.0 9 0 comm#8a5f93b2 38%
comm#fa4c2883 3 7 12.0 12.0 7 0 comm#8a5f93b2 49%
comm#da1fa6ff 13 2 11.0 11.0 2 0 comm#7e44bede 55%
comm#180cd039 5 8 10.0 10.0 8 0 comm#8a5f93b2 50%
comm#12f62659 6 4 9.0 9.0 4 0 comm#8a5f93b2 56%
comm#e1c397f1 8 3 9.0 9.0 3 0 comm#8a5f93b2 33%
comm#cfa272d9 15 6 8.0 8.0 6 0 comm#c233d180 75%
comm#0357d774 1 8 8.0 8.0 8 0 comm#8a5f93b2 37%
comm#d4e6d4f1 12 3 7.0 7.0 3 0 comm#8a5f93b2 29%
comm#eb6af029 15 1 6.1 6.1 1 0 comm#8a5f93b2 33%
comm#eb6af1dc 12 1 6.0 6.0 1 0 comm#8a5f93b2 33%
comm#dca24885 9 1 6.0 6.0 1 0 comm#8a5f93b2 100%
comm#288f4905 4 1 5.0 5.0 1 0 comm#8a5f93b2 100%
WORKLOAD pcpu-burst-16c-scx_p2dq
THERMAL/POWER
cpu temp peak 64.8 C avg 60.3 C
power avg 143.9 W peak 156.7 W
idle C2 avg 16.5% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
order-starved comm#5a4f434f passover_picks 3266 HIGH
kthread-strand comm#cfa272d9 max_strand_ms 130.003 HIGH
doublefree-logic 263196/0x000055b5846b8b50 bytes 24 HIGH
self-preempt tid=1826 stint_wake_ratio 10311 HIGH
doublefree-logic 263197/0x000055690fe19b50 bytes 24 HIGH
doublefree-logic 263195/0x00005614a1387b50 bytes 24 HIGH
self-preempt tid=1718 stint_wake_ratio 10269 HIGH
doublefree-logic 263194/0x0000557b3df19b50 bytes 24 HIGH
doublefree-logic 263193/0x000055e39f270b50 bytes 24 HIGH
self-preempt tid=1650 stint_wake_ratio 10205 HIGH
doublefree-logic 263192/0x000055e053c05b50 bytes 24 HIGH
doublefree-logic 263191/0x0000557fcf7c5b50 bytes 24 HIGH
self-preempt tid=1649 stint_wake_ratio 9718 HIGH
doublefree-logic 263190/0x0000561edcbaeb50 bytes 24 HIGH
order-starved comm#94672850 passover_picks 2023 HIGH
doublefree-logic 263189/0x0000562efb163b50 bytes 24 HIGH
self-preempt tid=1651 stint_wake_ratio 9660 HIGH
doublefree-logic 263188/0x000055771125ab50 bytes 24 HIGH
doublefree-logic 263187/0x0000558fd6aceb50 bytes 24 HIGH
self-preempt tid=1717 stint_wake_ratio 9320 HIGH
doublefree-logic 263186/0x0000561dbf60ab50 bytes 24 HIGH
doublefree-logic 263184/0x0000560621d45b50 bytes 24 HIGH
self-preempt tid=1827 stint_wake_ratio 9123 HIGH
doublefree-logic 263182/0x0000555c1a23eb50 bytes 24 HIGH
doublefree-logic 263181/0x000055b4b7e64b50 bytes 24 HIGH
order-starved comm#c233d180 passover_picks 1162 HIGH
doublefree-logic 263179/0x000055909cd9cb50 bytes 24 HIGH
self-preempt tid=1652 stint_wake_ratio 9039 HIGH
long-range-dependent dispatch-rate hurst 0.721131 MED
discord 21361ms profile_vs_mean 2.13577 MED
kthread-strand comm#fa4c2883 max_strand_ms 41.9963 MED
kthread-strand comm#da1fa8b2 max_strand_ms 34.9919 MED
kthread-strand comm#d4e6d4f1 max_strand_ms 34.9837 MED
kthread-strand comm#0357d927 max_strand_ms 32.9974 MED
kthread-strand comm#e3146c3c max_strand_ms 27.9988 MED
kthread-strand comm#1beaeefc max_strand_ms 24.9962 MED
kthread-strand comm#180cd039 max_strand_ms 23.9895 MED
kthread-strand comm#0878f8ff max_strand_ms 21.996 MED
kthread-strand comm#e1c397f1 max_strand_ms 21.9944 MED
kthread-strand comm#12f62659 max_strand_ms 21.9944 MED
kthread-strand comm#ca8154b4 max_strand_ms 21.994 MED
kthread-strand comm#ca81581a max_strand_ms 21.0086 MED
kthread-strand comm#dca24885 max_strand_ms 21.0005 MED
kthread-strand comm#ff850676 max_strand_ms 20.9996 MED
kthread-strand comm#0dbd5298 max_strand_ms 20.9828 MED
kthread-strand comm#1beaf0af max_strand_ms 20.0043 MED
kthread-strand comm#0357d774 max_strand_ms 20 MED
kthread-strand comm#493ffe92 max_strand_ms 19.9944 MED
kthread-strand comm#c43b8240 max_strand_ms 18.0013 MED
kthread-strand comm#288f4905 max_strand_ms 17.9863 MED
kthread-strand comm#fa4c3102 max_strand_ms 17.0012 MED
kthread-strand comm#180cccd3 max_strand_ms 16.997 MED
kthread-strand comm#ddbfab61 max_strand_ms 16.738 MED
kthread-strand comm#1beaf415 max_strand_ms 15.9988 MED
kthread-strand comm#288f459f max_strand_ms 13.9908 MED
kthread-strand comm#cfa26f73 max_strand_ms 12.7279 MED
kthread-strand comm#e1c3948b max_strand_ms 10.0059 MED
poll-slow 263166/pandemonium duration_ms 0.00164 LOW
hot-cpu 2 l2_miss_share 7.20434 LOW
poll-slow 263167/pandemonium duration_ms 0.00145 LOW
poll-slow 263173/pandemonium duration_ms 0.00133 LOW
poll-slow 263172/pandemonium duration_ms 0.00133 LOW
poll-slow 263171/pandemonium duration_ms 0.0013 LOW
CAPTURE complete -- 394455 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 57.5k wake2run; p50 2998us p99 28000us p999 42005us worst 68000us; 26.3% fast(<100us) / 1.3% mid / 72.4% tick-floor(>=900us); 0.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~57512 distinct values; inversion ratio 0.41
REPORT dispatch-stall
VERDICT: 41.6k saturated wakes over the 900us floor; PREEMPT-STARVED 0% (0 intervening picks) / ORDER-STARVED 100% (CPU served others first); avg 3.4 pass-overs, p99 14 pass-overs
REPORT kstrand
VERDICT: 558 strands across 28 per-CPU kthreads; worst HELD strand 130.0ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#cfa272d9 15 22 130.0 130.0 22 0 comm#8a5f93b2 30%
comm#fa4c2883 3 58 42.0 42.0 58 0 comm#8a5f93b2 50%
comm#da1fa8b2 13 18 35.0 35.0 18 0 comm#8a5f93b2 60%
comm#d4e6d4f1 12 58 35.0 35.0 58 0 comm#8a5f93b2 60%
comm#0357d927 1 24 33.0 33.0 24 0 comm#8a5f93b2 64%
comm#e3146c3c 11 14 28.0 28.0 14 0 comm#8a5f93b2 75%
comm#1beaeefc 4 32 25.0 25.0 32 0 comm#8a5f93b2 84%
comm#180cd039 5 25 24.0 24.0 25 0 comm#8a5f93b2 88%
comm#0878f8ff 0 114 22.0 21.0 114 0 comm#8a5f93b2 95%
comm#e1c397f1 8 36 22.0 22.0 36 0 comm#8a5f93b2 95%
comm#12f62659 6 30 22.0 22.0 30 0 comm#8a5f93b2 95%
comm#ca8154b4 14 31 22.0 22.0 31 0 comm#8a5f93b2 95%
comm#ca81581a 14 5 21.0 21.0 5 0 comm#8a5f93b2 100%
comm#dca24885 9 14 21.0 21.0 14 0 comm#8a5f93b2 100%
comm#ff850676 2 15 21.0 21.0 15 0 comm#8a5f93b2 100%
comm#0dbd5298 7 20 21.0 21.0 20 0 comm#8a5f93b2 100%
comm#1beaf0af 4 6 20.0 20.0 6 0 comm#8a5f93b2 100%
comm#0357d774 1 4 20.0 20.0 4 0 comm#8a5f93b2 100%
comm#493ffe92 10 15 20.0 20.0 15 0 comm#8a5f93b2 100%
comm#c43b8240 12 3 18.0 18.0 3 0 comm#8a5f93b2 100%
WORKLOAD pcpu-burst-16c-scx_rustland
THERMAL/POWER
cpu temp peak 62.5 C avg 57.4 C
power avg 126.5 W peak 158.6 W
idle C2 avg 42.2% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
order-starved comm#4fc5a1ab passover_picks 29294 HIGH
doublefree-logic 262115/0x000056333d3b7b50 bytes 24 HIGH
self-preempt tid=262066 stint_wake_ratio 41484 HIGH
doublefree-logic 262114/0x00005619ab7c4b50 bytes 24 HIGH
doublefree-logic 262113/0x000055a168d7cb50 bytes 24 HIGH
self-preempt tid=1717 stint_wake_ratio 8435 HIGH
doublefree-logic 262112/0x000055a7843fab50 bytes 24 HIGH
doublefree-logic 262111/0x0000561751621b50 bytes 24 HIGH
self-preempt tid=1651 stint_wake_ratio 8428 HIGH
doublefree-logic 262110/0x0000560341f26b50 bytes 24 HIGH
doublefree-logic 262109/0x0000558aab152b50 bytes 24 HIGH
self-preempt tid=1650 stint_wake_ratio 8427 HIGH
doublefree-logic 262108/0x0000560446bfab50 bytes 24 HIGH
order-starved comm#5a4f434f passover_picks 3659 HIGH
doublefree-logic 262107/0x000055b89bb70b50 bytes 24 HIGH
self-preempt tid=1649 stint_wake_ratio 8391 HIGH
doublefree-logic 262106/0x000055d952090b50 bytes 24 HIGH
doublefree-logic 262104/0x00005584be80eb50 bytes 24 HIGH
self-preempt tid=1652 stint_wake_ratio 8350 HIGH
doublefree-logic 262103/0x0000561a1658ab50 bytes 24 HIGH
doublefree-logic 262101/0x00005651f308ab50 bytes 24 HIGH
self-preempt tid=1718 stint_wake_ratio 8254 HIGH
doublefree-logic 262099/0x000055a7a8c73b50 bytes 24 HIGH
doublefree-logic 262098/0x00005600062e4b50 bytes 24 HIGH
order-starved comm#328ab0c6 passover_picks 2299 HIGH
doublefree-logic 262097/0x000055d23b9cab50 bytes 24 HIGH
self-preempt tid=1827 stint_wake_ratio 8192 HIGH
kthread-strand comm#1beaf0af max_strand_ms 20.9955 MED
long-range-dependent dispatch-rate hurst 0.827385 MED
discord 14937ms profile_vs_mean 1.61413 MED
kthread-strand comm#fa4c2883 max_strand_ms 20.9946 MED
kthread-strand comm#288f4752 max_strand_ms 20.9923 MED
kthread-strand comm#288f4905 max_strand_ms 20.992 MED
kthread-strand comm#180cd039 max_strand_ms 18.9953 MED
kthread-strand comm#cfa272d9 max_strand_ms 14.9929 MED
kthread-strand comm#0dbd5298 max_strand_ms 11.9973 MED
kthread-strand comm#da1fa8b2 max_strand_ms 10.0046 MED
kthread-strand comm#ca8154b4 max_strand_ms 9.99559 MED
kthread-strand comm#ff850676 max_strand_ms 8.00294 MED
kthread-strand comm#e3146c3c max_strand_ms 7.99161 MED
mean-reverting migration-rate hurst -2.33096 MED
poll-slow 262090/pandemonium duration_ms 0.00175 LOW
hot-cpu 5 l2_miss_share 7.12633 LOW
poll-slow 262084/pandemonium duration_ms 0.0017 LOW
poll-slow 262085/pandemonium duration_ms 0.00143 LOW
poll-slow 262086/pandemonium duration_ms 0.00142 LOW
poll-slow 262094/pandemonium duration_ms 0.00128 LOW
CAPTURE complete -- 334236 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 17.1k wake2run; p50 1688us p99 152000us p999 3281002us worst 5662551us; 23.9% fast(<100us) / 10.3% mid / 65.8% tick-floor(>=900us); 0.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~17140 distinct values; inversion ratio 0.43
REPORT dispatch-stall
VERDICT: 11.3k saturated wakes over the 900us floor; PREEMPT-STARVED 0% (0 intervening picks) / ORDER-STARVED 100% (CPU served others first); avg 13.8 pass-overs, p99 75 pass-overs
REPORT kstrand
VERDICT: 11 strands across 11 per-CPU kthreads; worst HELD strand 21.0ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#1beaf0af 4 1 21.0 21.0 1 0 comm#8a5f93b2 100%
comm#fa4c2883 3 1 21.0 21.0 1 0 comm#8a5f93b2 100%
comm#288f4752 5 1 21.0 21.0 1 0 comm#8a5f93b2 100%
comm#288f4905 4 1 21.0 21.0 1 0 comm#8a5f93b2 100%
comm#180cd039 5 1 19.0 19.0 1 0 comm#8a5f93b2 100%
comm#cfa272d9 15 1 15.0 15.0 1 0 comm#8a5f93b2 100%
comm#0dbd5298 7 1 12.0 12.0 1 0 comm#8a5f93b2 100%
comm#da1fa8b2 13 1 10.0 10.0 1 0 comm#8a5f93b2 100%
comm#ca8154b4 14 1 10.0 10.0 1 0 comm#8a5f93b2 100%
comm#ff850676 2 1 8.0 8.0 1 0 comm#8a5f93b2 100%
comm#e3146c3c 11 1 8.0 8.0 1 0 comm#8a5f93b2 100%
WORKLOAD pcpu-burst-16c-scx_rusty
THERMAL/POWER
cpu temp peak 63.5 C avg 59.1 C
power avg 138.0 W peak 159.9 W
idle C2 avg 28.7% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
order-starved comm#5a4f434f passover_picks 3671 HIGH
doublefree-logic 258475/0x0000557f85c0cb50 bytes 24 HIGH
self-preempt tid=1827 stint_wake_ratio 12282 HIGH
doublefree-logic 258474/0x000055dc6064bb50 bytes 24 HIGH
doublefree-logic 258473/0x00005567d81f2b50 bytes 24 HIGH
self-preempt tid=1718 stint_wake_ratio 12281 HIGH
doublefree-logic 258472/0x000055fc0ced0b50 bytes 24 HIGH
doublefree-logic 258471/0x00005560141cdb50 bytes 24 HIGH
self-preempt tid=1649 stint_wake_ratio 12279 HIGH
doublefree-logic 258470/0x0000562e1598ab50 bytes 24 HIGH
doublefree-logic 258469/0x0000557ad8695b50 bytes 24 HIGH
self-preempt tid=1651 stint_wake_ratio 12279 HIGH
doublefree-logic 258466/0x0000559084668b50 bytes 24 HIGH
order-starved comm#328ab0c6 passover_picks 2053 HIGH
doublefree-logic 258468/0x0000562c2aa9eb50 bytes 24 HIGH
self-preempt tid=1826 stint_wake_ratio 11740 HIGH
doublefree-logic 258467/0x0000560ff0cfdb50 bytes 24 HIGH
doublefree-logic 258465/0x00005609cf075b50 bytes 24 HIGH
self-preempt tid=1652 stint_wake_ratio 11740 HIGH
doublefree-logic 258464/0x00005586063abb50 bytes 24 HIGH
doublefree-logic 258463/0x000055bc19fc2b50 bytes 24 HIGH
self-preempt tid=1650 stint_wake_ratio 11737 HIGH
doublefree-logic 258459/0x000055a6661bfb50 bytes 24 HIGH
doublefree-logic 258457/0x000055a8b3ee9b50 bytes 24 HIGH
order-starved comm#328ab0c6 passover_picks 2019 HIGH
doublefree-logic 258456/0x00005555f7449b50 bytes 24 HIGH
self-preempt tid=1717 stint_wake_ratio 11734 HIGH
kthread-strand comm#0dbd5298 max_strand_ms 42.995 MED
long-range-dependent dispatch-rate hurst 0.817273 MED
discord 18014ms profile_vs_mean 1.7859 MED
kthread-strand comm#0878f233 max_strand_ms 35.9957 MED
kthread-strand comm#0878f8ff max_strand_ms 34.9955 MED
kthread-strand comm#d4e6d4f1 max_strand_ms 30.9947 MED
kthread-strand comm#dca24885 max_strand_ms 29.9948 MED
kthread-strand comm#180cd039 max_strand_ms 24.9923 MED
kthread-strand comm#e1c397f1 max_strand_ms 22.994 MED
kthread-strand comm#da1fa6ff max_strand_ms 22.9928 MED
kthread-strand comm#493ffe92 max_strand_ms 21.9994 MED
kthread-strand comm#0357d774 max_strand_ms 21.9957 MED
kthread-strand comm#ca8154b4 max_strand_ms 21.9916 MED
kthread-strand comm#fa4c2883 max_strand_ms 21.9904 MED
kthread-strand comm#288f4086 max_strand_ms 20.9991 MED
kthread-strand comm#12f62659 max_strand_ms 20.9976 MED
kthread-strand comm#e3146c3c max_strand_ms 20.9966 MED
kthread-strand comm#0dbd55fe max_strand_ms 20.9925 MED
kthread-strand comm#288f43ec max_strand_ms 20.9908 MED
kthread-strand comm#288f4752 max_strand_ms 20.9897 MED
kthread-strand comm#ff850676 max_strand_ms 20.004 MED
kthread-strand comm#cfa26f73 max_strand_ms 19.9961 MED
kthread-strand comm#1beaf0af max_strand_ms 19.9936 MED
kthread-strand comm#288f3ed3 max_strand_ms 19.9871 MED
kthread-strand comm#288f4239 max_strand_ms 18.9846 MED
kthread-strand comm#e1c392d8 max_strand_ms 16.9983 MED
kthread-strand comm#288f4905 max_strand_ms 16.985 MED
kthread-strand comm#288f459f max_strand_ms 15.9754 MED
kthread-strand comm#da1faa65 max_strand_ms 14.9962 MED
kthread-strand comm#288f3d20 max_strand_ms 14.9879 MED
kthread-strand comm#cfa272d9 max_strand_ms 12.2767 MED
kthread-strand comm#0357d927 max_strand_ms 10.0146 MED
poll-slow 258444/pandemonium duration_ms 0.00146 LOW
hot-cpu 2 l2_miss_share 6.86171 LOW
poll-slow 258448/pandemonium duration_ms 0.00138 LOW
poll-slow 258451/pandemonium duration_ms 0.00135 LOW
poll-slow 258447/pandemonium duration_ms 0.00133 LOW
poll-slow 258449/pandemonium duration_ms 0.00132 LOW
CAPTURE complete -- 318870 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 9.1k wake2run; p50 10us p99 704998us p999 1975003us worst 3413999us; 63.9% fast(<100us) / 2.5% mid / 33.6% tick-floor(>=900us); 0.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~9139 distinct values; inversion ratio 0.38
REPORT dispatch-stall
VERDICT: 3.1k saturated wakes over the 900us floor; PREEMPT-STARVED 0% (0 intervening picks) / ORDER-STARVED 100% (CPU served others first); avg 59.3 pass-overs, p99 1028 pass-overs
REPORT kstrand
VERDICT: 444 strands across 30 per-CPU kthreads; worst HELD strand 43.0ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#0dbd5298 7 42 43.0 43.0 42 0 comm#8a5f93b2 51%
comm#0878f233 0 32 36.0 36.0 32 0 comm#8a5f93b2 58%
comm#0878f8ff 0 37 35.0 35.0 37 0 comm#8a5f93b2 94%
comm#d4e6d4f1 12 22 31.0 31.0 22 0 comm#8a5f93b2 68%
comm#dca24885 9 10 30.0 30.0 10 0 comm#94672850 70%
comm#180cd039 5 27 25.0 25.0 27 0 comm#8a5f93b2 100%
comm#e1c397f1 8 35 23.0 23.0 35 0 comm#8a5f93b2 91%
comm#da1fa6ff 13 25 23.0 23.0 25 0 comm#8a5f93b2 91%
comm#493ffe92 10 21 22.0 22.0 21 0 comm#8a5f93b2 95%
comm#0357d774 1 29 22.0 22.0 29 0 comm#8a5f93b2 95%
comm#ca8154b4 14 17 22.0 22.0 17 0 comm#8a5f93b2 100%
comm#fa4c2883 3 14 22.0 22.0 14 0 comm#8a5f93b2 100%
comm#288f4086 1 5 21.0 21.0 5 0 comm#8a5f93b2 100%
comm#12f62659 6 13 21.0 21.0 13 0 comm#8a5f93b2 100%
comm#e3146c3c 11 10 21.0 21.0 10 0 comm#8a5f93b2 100%
comm#0dbd55fe 7 10 21.0 21.0 10 0 comm#8a5f93b2 100%
comm#288f43ec 7 2 21.0 21.0 2 0 comm#8a5f93b2 100%
comm#288f4752 5 4 21.0 21.0 4 0 comm#8a5f93b2 100%
comm#ff850676 2 23 20.0 20.0 23 0 comm#8a5f93b2 100%
comm#cfa26f73 15 15 20.0 20.0 15 0 comm#c233d180 100%
WORKLOAD pcpu-burst-16c-scx_tickless
THERMAL/POWER
cpu temp peak 63.0 C avg 60.1 C
power avg 143.5 W peak 161.3 W
idle C2 avg 20.6% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
order-starved comm#5a4f434f passover_picks 3208 HIGH
doublefree-logic 264287/0x0000561cc7eecb50 bytes 24 HIGH
self-preempt tid=1651 stint_wake_ratio 9180 HIGH
doublefree-logic 264286/0x0000559db3c13b50 bytes 24 HIGH
doublefree-logic 264285/0x000055af02c48b50 bytes 24 HIGH
self-preempt tid=1650 stint_wake_ratio 9177 HIGH
doublefree-logic 264284/0x00005606da2b0b50 bytes 24 HIGH
doublefree-logic 264283/0x0000562981415b50 bytes 24 HIGH
self-preempt tid=1717 stint_wake_ratio 9176 HIGH
doublefree-logic 264282/0x000055e806a39b50 bytes 24 HIGH
doublefree-logic 264281/0x00005581eb385b50 bytes 24 HIGH
self-preempt tid=1827 stint_wake_ratio 9175 HIGH
doublefree-logic 264280/0x000055872e651b50 bytes 24 HIGH
order-starved comm#328ab0c6 passover_picks 895 HIGH
doublefree-logic 264279/0x000056160d7f2b50 bytes 24 HIGH
self-preempt tid=1718 stint_wake_ratio 9174 HIGH
doublefree-logic 264277/0x000055cc6bb05b50 bytes 24 HIGH
doublefree-logic 264276/0x000055983db7eb50 bytes 24 HIGH
self-preempt tid=1826 stint_wake_ratio 9174 HIGH
doublefree-logic 264274/0x0000556efeaf2b50 bytes 24 HIGH
doublefree-logic 264272/0x0000556c8a150b50 bytes 24 HIGH
self-preempt tid=1649 stint_wake_ratio 9172 HIGH
doublefree-logic 264270/0x000055932dc48b50 bytes 24 HIGH
doublefree-logic 264267/0x0000564148e50b50 bytes 24 HIGH
order-starved comm#c233d180 passover_picks 843 HIGH
doublefree-logic 264266/0x000055ab39cd7b50 bytes 24 HIGH
self-preempt tid=1652 stint_wake_ratio 9168 HIGH
kthread-strand comm#cfa26f73 max_strand_ms 38.3903 MED
long-range-dependent dispatch-rate hurst 0.70845 MED
discord 6868ms profile_vs_mean 1.63515 MED
kthread-strand comm#da1fa8b2 max_strand_ms 37.9955 MED
kthread-strand comm#ff850676 max_strand_ms 35.9946 MED
kthread-strand comm#0357d927 max_strand_ms 35.994 MED
kthread-strand comm#dca24885 max_strand_ms 34.9947 MED
kthread-strand comm#0878f8ff max_strand_ms 31.9948 MED
kthread-strand comm#0dbd5298 max_strand_ms 31.9933 MED
kthread-strand comm#180cd039 max_strand_ms 30.9974 MED
kthread-strand comm#288f4752 max_strand_ms 29.9809 MED
kthread-strand comm#ca8154b4 max_strand_ms 26.9956 MED
kthread-strand comm#fa4c2883 max_strand_ms 25 MED
kthread-strand comm#e1c397f1 max_strand_ms 21.127 MED
kthread-strand comm#288f3ed3 max_strand_ms 21.001 MED
kthread-strand comm#12f62659 max_strand_ms 20.9954 MED
kthread-strand comm#1beaeefc max_strand_ms 20.9947 MED
kthread-strand comm#288f4fd1 max_strand_ms 20.9805 MED
kthread-strand comm#288f43ec max_strand_ms 20.8464 MED
kthread-strand comm#d4e6d4f1 max_strand_ms 19.9975 MED
kthread-strand comm#e3146c3c max_strand_ms 19.9952 MED
kthread-strand comm#493ffe92 max_strand_ms 19.9942 MED
kthread-strand comm#288f4905 max_strand_ms 19.9783 MED
kthread-strand comm#288f4e1e max_strand_ms 19.972 MED
kthread-strand comm#eb6af029 max_strand_ms 18.9736 MED
kthread-strand comm#eb6af542 max_strand_ms 18.9243 MED
kthread-strand comm#ddbfab61 max_strand_ms 14.9991 MED
kthread-strand comm#288f4086 max_strand_ms 11.9481 MED
kthread-strand comm#eb6af1dc max_strand_ms 9.98663 MED
kthread-strand comm#eb6af38f max_strand_ms 8.97964 MED
kthread-strand comm#288f4239 max_strand_ms 8.97555 MED
kthread-strand comm#288f459f max_strand_ms 8.96299 MED
kthread-strand comm#cfa272d9 max_strand_ms 7.02272 MED
mean-reverting migration-rate hurst -2.20147 MED
poll-slow 264264/pandemonium duration_ms 0.00187 LOW
hot-cpu 2 l2_miss_share 7.00168 LOW
poll-slow 264260/pandemonium duration_ms 0.0013 LOW
poll-slow 264269/pandemonium duration_ms 0.00128 LOW
poll-slow 264261/pandemonium duration_ms 0.00128 LOW
poll-slow 264256/pandemonium duration_ms 0.00126 LOW
CAPTURE complete -- 191317 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 1.7k wake2run; p50 18us p99 2337101us p999 2414996us worst 2444071us; 74.1% fast(<100us) / 10.2% mid / 15.7% tick-floor(>=900us); 0.5% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~1681 distinct values; inversion ratio 0.39
REPORT dispatch-stall
VERDICT: 269 saturated wakes over the 900us floor; PREEMPT-STARVED 0% (0 intervening picks) / ORDER-STARVED 100% (CPU served others first); avg 319.1 pass-overs, p99 1860 pass-overs
REPORT kstrand
VERDICT: 781 strands across 31 per-CPU kthreads; worst HELD strand 38.4ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#cfa26f73 15 41 38.4 38.4 41 0 comm#7e44bede 55%
comm#da1fa8b2 13 36 38.0 38.0 36 0 comm#7e44bede 55%
comm#ff850676 2 57 36.0 36.0 57 0 comm#7e44bede 58%
comm#0357d927 1 41 36.0 36.0 41 0 comm#7e44bede 58%
comm#dca24885 9 40 35.0 35.0 40 0 comm#7e44bede 60%
comm#0878f8ff 0 79 32.0 32.0 79 0 comm#7e44bede 66%
comm#0dbd5298 7 31 32.0 32.0 31 0 comm#7e44bede 66%
comm#180cd039 5 37 31.0 31.0 37 0 comm#7e44bede 68%
comm#288f4752 5 2 30.0 30.0 2 0 comm#7e44bede 69%
comm#ca8154b4 14 48 27.0 27.0 48 0 comm#7e44bede 78%
comm#fa4c2883 3 48 25.0 25.0 48 0 comm#7e44bede 84%
comm#e1c397f1 8 81 21.1 21.1 81 0 comm#8a5f93b2 99%
comm#288f3ed3 2 3 21.0 21.0 3 0 comm#f7d5fbc2 100%
comm#12f62659 6 41 21.0 21.0 41 0 comm#8a5f93b2 100%
comm#1beaeefc 4 44 21.0 21.0 44 0 comm#7e44bede 100%
comm#288f4fd1 8 3 21.0 21.0 3 0 comm#8a5f93b2 100%
comm#288f43ec 7 4 20.8 20.8 4 0 comm#8a5f93b2 100%
comm#d4e6d4f1 12 46 20.0 20.0 46 0 comm#8a5f93b2 100%
comm#e3146c3c 11 27 20.0 20.0 27 0 comm#8a5f93b2 100%
comm#493ffe92 10 38 20.0 20.0 38 0 comm#8a5f93b2 100%
WORKLOAD sojourn-pressure-16c-EEVDF
THERMAL/POWER
cpu temp peak 60.2 C avg 58.8 C
power avg 137.4 W peak 157.5 W
idle C2 avg 28.6% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
order-starved comm#ff2dc3b5 passover_picks 363 HIGH
held-cpu comm#8a5f93b2 held_ms 13.3223 HIGH
self-preempt tid=1649 stint_wake_ratio 12863 HIGH
self-preempt tid=1827 stint_wake_ratio 11222 HIGH
self-preempt tid=1718 stint_wake_ratio 11096 HIGH
self-preempt tid=1717 stint_wake_ratio 10484 HIGH
order-starved comm#8a5f93b2 passover_picks 333 HIGH
held-cpu comm#8a5f93b2 held_ms 11.4256 HIGH
self-preempt tid=1826 stint_wake_ratio 10441 HIGH
self-preempt tid=1650 stint_wake_ratio 9654 HIGH
self-preempt tid=1652 stint_wake_ratio 9618 HIGH
order-starved comm#8a5f93b2 passover_picks 318 HIGH
held-cpu comm#8a5f93b2 held_ms 9.40304 HIGH
self-preempt tid=1651 stint_wake_ratio 9336 HIGH
mean-reverting migration-rate hurst -1.47101 MED
discord 7056ms profile_vs_mean 1.54771 MED
hot-cpu 15 l2_miss_share 10.2672 MED
CAPTURE complete -- 381138 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 2.2k wake2run; p50 721us p99 4284us p999 4978us worst 5383us; 42.5% fast(<100us) / 9.2% mid / 48.3% tick-floor(>=900us); 0.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~2097 distinct values; inversion ratio 0.52
REPORT dispatch-stall
VERDICT: 1.1k saturated wakes over the 900us floor; PREEMPT-STARVED 3% (0 intervening picks) / ORDER-STARVED 97% (CPU served others first); avg 2.8 pass-overs, p99 7 pass-overs
WORKLOAD sojourn-pressure-16c-PANDEMONIUM
THERMAL/POWER
cpu temp peak 60.5 C avg 57.9 C
power avg 134.9 W peak 159.7 W
idle C2 avg 34.3% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
held-cpu comm#8a5f93b2 held_ms 3.74345 HIGH
self-preempt tid=1827 stint_wake_ratio 13022 HIGH
self-preempt tid=1717 stint_wake_ratio 13001 HIGH
self-preempt tid=1652 stint_wake_ratio 12977 HIGH
self-preempt tid=1826 stint_wake_ratio 12927 HIGH
held-cpu comm#8a5f93b2 held_ms 2.01778 HIGH
self-preempt tid=1718 stint_wake_ratio 12926 HIGH
self-preempt tid=1650 stint_wake_ratio 12926 HIGH
self-preempt tid=1649 stint_wake_ratio 12884 HIGH
held-cpu comm#8a5f93b2 held_ms 1.94015 HIGH
self-preempt tid=1651 stint_wake_ratio 12875 HIGH
order-starved comm#dd98ba5b passover_picks 6 MED
mean-reverting dispatch-rate hurst 0.301 MED
hot-cpu 4 l2_miss_share 9.37837 MED
order-starved comm#811e12bd passover_picks 3 MED
order-starved comm#8a5f93b2 passover_picks 2 MED
CAPTURE complete -- 497666 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 4.4k wake2run; p50 4us p99 872us p999 1655us worst 2640us; 95.2% fast(<100us) / 4.0% mid / 0.8% tick-floor(>=900us); 0.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~3625 distinct values; inversion ratio 0.55
REPORT dispatch-stall
VERDICT: 37 saturated wakes over the 900us floor; PREEMPT-STARVED 65% (0 intervening picks) / ORDER-STARVED 35% (CPU served others first); avg 3.8 pass-overs, p99 14 pass-overs
WORKLOAD sojourn-pressure-16c-scx_beerland
THERMAL/POWER
cpu temp peak 61.2 C avg 58.2 C
power avg 141.5 W peak 159.1 W
idle C2 avg 23.3% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
order-starved comm#328ab0c6 passover_picks 7 HIGH
self-preempt tid=1651 stint_wake_ratio 13298 HIGH
self-preempt tid=1649 stint_wake_ratio 13297 HIGH
self-preempt tid=1650 stint_wake_ratio 13294 HIGH
self-preempt tid=1826 stint_wake_ratio 13293 HIGH
order-starved comm#328ab0c6 passover_picks 5 HIGH
self-preempt tid=1718 stint_wake_ratio 13284 HIGH
self-preempt tid=1652 stint_wake_ratio 13281 HIGH
self-preempt tid=1827 stint_wake_ratio 13278 HIGH
order-starved comm#328ab0c6 passover_picks 4 HIGH
self-preempt tid=1717 stint_wake_ratio 13271 HIGH
kthread-strand comm#0878f080 max_strand_ms 14.01 MED
long-range-dependent migration-rate hurst 0.692562 MED
kthread-strand comm#2f099b5a max_strand_ms 12.9994 MED
kthread-strand comm#ddbfab61 max_strand_ms 11.9734 MED
kthread-strand comm#658123d8 max_strand_ms 11.1306 MED
kthread-strand comm#d1030141 max_strand_ms 8.99997 MED
mean-reverting dispatch-rate hurst 0.302278 MED
hot-cpu 15 l2_miss_share 8.93081 LOW
CAPTURE complete -- 361382 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 4.5k wake2run; p50 2us p99 10us p999 2756us worst 7174us; 99.8% fast(<100us) / 0.0% mid / 0.1% tick-floor(>=900us); 0.8% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~2970 distinct values; inversion ratio 0.52
REPORT dispatch-stall
VERDICT: 6 saturated wakes over the 900us floor; PREEMPT-STARVED 0% (0 intervening picks) / ORDER-STARVED 100% (CPU served others first); avg 11.8 pass-overs, p99 44 pass-overs
REPORT kstrand
VERDICT: 9 strands across 5 per-CPU kthreads; worst HELD strand 14.0ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#0878f080 0 1 14.0 14.0 1 0 comm#8a5f93b2 12%
comm#2f099b5a 2 1 13.0 13.0 1 0 comm#f7d5fbc2 23%
comm#ddbfab61 5 2 12.0 12.0 2 0 comm#8a5f93b2 33%
comm#658123d8 0 4 11.1 11.1 4 0 comm#8a5f93b2 19%
comm#d1030141 15 1 9.0 9.0 1 0 comm#8a5f93b2 22%
WORKLOAD sojourn-pressure-16c-scx_bpfland
THERMAL/POWER
cpu temp peak 60.8 C avg 57.5 C
power avg 136.3 W peak 157.6 W
idle C2 avg 29.3% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
held-cpu comm#8a5f93b2 held_ms 38.2326 HIGH
self-preempt tid=1718 stint_wake_ratio 9244 HIGH
self-preempt tid=1649 stint_wake_ratio 9240 HIGH
self-preempt tid=1651 stint_wake_ratio 9238 HIGH
self-preempt tid=1717 stint_wake_ratio 9238 HIGH
held-cpu comm#8a5f93b2 held_ms 35.486 HIGH
self-preempt tid=1650 stint_wake_ratio 9238 HIGH
self-preempt tid=1826 stint_wake_ratio 9238 HIGH
self-preempt tid=1827 stint_wake_ratio 9236 HIGH
held-cpu comm#8a5f93b2 held_ms 34.5044 HIGH
self-preempt tid=1652 stint_wake_ratio 9234 HIGH
order-starved comm#5a4f434f passover_picks 1 MED
kthread-strand comm#cfa272d9 max_strand_ms 37.0007 MED
mean-reverting migration-rate hurst 0.46758 MED
kthread-strand comm#ff850676 max_strand_ms 12.0007 MED
kthread-strand comm#1beaeefc max_strand_ms 6.00442 MED
kthread-strand comm#fa4c3102 max_strand_ms 5.08657 MED
kthread-strand comm#d4e6d4f1 max_strand_ms 5.04868 MED
mean-reverting dispatch-rate hurst 0.396001 MED
hot-cpu 15 l2_miss_share 8.27512 LOW
CAPTURE complete -- 346778 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 2.7k wake2run; p50 930us p99 943us p999 950us worst 954us; 13.7% fast(<100us) / 21.6% mid / 64.7% tick-floor(>=900us); 49.3% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~2686 distinct values; inversion ratio 0.48
REPORT dispatch-stall
VERDICT: 1.7k saturated wakes over the 900us floor; PREEMPT-STARVED 100% (0 intervening picks) / ORDER-STARVED 0% (CPU served others first); avg 1.0 pass-overs, p99 0 pass-overs
REPORT kstrand
VERDICT: 13 strands across 5 per-CPU kthreads; worst HELD strand 37.0ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#cfa272d9 15 6 37.0 37.0 6 0 comm#c233d180 51%
comm#ff850676 2 2 12.0 12.0 2 0 comm#f7d5fbc2 58%
comm#1beaeefc 4 2 6.0 6.0 2 0 comm#8a5f93b2 17%
comm#fa4c3102 3 1 5.1 5.1 1 0 comm#8a5f93b2 20%
comm#d4e6d4f1 12 2 5.0 5.0 2 0 comm#8a5f93b2 20%
WORKLOAD sojourn-pressure-16c-scx_cake
THERMAL/POWER
cpu temp peak 61.0 C avg 58.5 C
power avg 133.0 W peak 156.9 W
idle C2 avg 37.1% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
order-starved comm#328ab0c6 passover_picks 171 HIGH
held-cpu comm#8a5f93b2 held_ms 2.82991 HIGH
self-preempt tid=1652 stint_wake_ratio 13069 HIGH
self-preempt tid=1651 stint_wake_ratio 13065 HIGH
self-preempt tid=1827 stint_wake_ratio 6066 HIGH
self-preempt tid=1649 stint_wake_ratio 6014 HIGH
order-starved comm#5a4f434f passover_picks 129 HIGH
held-cpu comm#8a5f93b2 held_ms 1.00371 HIGH
self-preempt tid=1718 stint_wake_ratio 5998 HIGH
self-preempt tid=1826 stint_wake_ratio 5985 HIGH
self-preempt tid=1650 stint_wake_ratio 5983 HIGH
order-starved comm#328ab0c6 passover_picks 43 HIGH
held-cpu comm#8a5f93b2 held_ms 0.967011 HIGH
self-preempt tid=1717 stint_wake_ratio 5981 HIGH
kthread-strand comm#ff850676 max_strand_ms 32.9995 MED
discord 11011ms profile_vs_mean 1.67032 MED
hot-cpu 0 l2_miss_share 13.7189 MED
kthread-strand comm#1beaeefc max_strand_ms 25.9991 MED
kthread-strand comm#180cd039 max_strand_ms 21.9989 MED
kthread-strand comm#dca24885 max_strand_ms 20.9996 MED
kthread-strand comm#0878f080 max_strand_ms 20.9994 MED
kthread-strand comm#e3146c3c max_strand_ms 20.9994 MED
kthread-strand comm#12f62659 max_strand_ms 20.9993 MED
kthread-strand comm#fa4c3102 max_strand_ms 19.9996 MED
kthread-strand comm#da1fa8b2 max_strand_ms 17.9998 MED
kthread-strand comm#0357d927 max_strand_ms 16.9998 MED
kthread-strand comm#d4e6d4f1 max_strand_ms 15.9996 MED
kthread-strand comm#493ffe92 max_strand_ms 14.9999 MED
kthread-strand comm#ca8154b4 max_strand_ms 12.9914 MED
kthread-strand comm#0dbd57b1 max_strand_ms 11.5576 MED
kthread-strand comm#e1c397f1 max_strand_ms 10.9959 MED
kthread-strand comm#cfa26dc0 max_strand_ms 7.00006 MED
CAPTURE complete -- 252964 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 1.6k wake2run; p50 6us p99 12937us p999 14965us worst 16931us; 62.2% fast(<100us) / 3.4% mid / 34.4% tick-floor(>=900us); 0.0% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~1448 distinct values; inversion ratio 0.87
REPORT dispatch-stall
VERDICT: 559 saturated wakes over the 900us floor; PREEMPT-STARVED 1% (0 intervening picks) / ORDER-STARVED 99% (CPU served others first); avg 11.1 pass-overs, p99 24 pass-overs
REPORT kstrand
VERDICT: 80 strands across 16 per-CPU kthreads; worst HELD strand 33.0ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#ff850676 2 5 33.0 33.0 5 0 comm#8a5f93b2 64%
comm#1beaeefc 4 4 26.0 26.0 4 0 comm#8a5f93b2 81%
comm#180cd039 5 5 22.0 22.0 5 0 comm#8a5f93b2 95%
comm#dca24885 9 5 21.0 21.0 5 0 comm#8a5f93b2 100%
comm#0878f080 0 16 21.0 21.0 16 0 comm#8a5f93b2 100%
comm#e3146c3c 11 6 21.0 21.0 6 0 comm#8a5f93b2 97%
comm#12f62659 6 10 21.0 21.0 10 0 comm#8a5f93b2 100%
comm#fa4c3102 3 6 20.0 20.0 6 0 comm#8a5f93b2 100%
comm#da1fa8b2 13 5 18.0 18.0 5 0 comm#8a5f93b2 100%
comm#0357d927 1 7 17.0 17.0 7 0 comm#8a5f93b2 100%
comm#d4e6d4f1 12 1 16.0 16.0 1 0 comm#8a5f93b2 100%
comm#493ffe92 10 3 15.0 15.0 3 0 comm#8a5f93b2 100%
comm#ca8154b4 14 1 13.0 13.0 1 0 comm#8a5f93b2 87%
comm#0dbd57b1 7 3 11.6 11.6 3 0 comm#94672850 100%
comm#e1c397f1 8 1 11.0 11.0 1 0 comm#8a5f93b2 100%
comm#cfa26dc0 15 2 7.0 7.0 2 0 comm#8a5f93b2 100%
WORKLOAD sojourn-pressure-16c-scx_cosmos
THERMAL/POWER
cpu temp peak 61.2 C avg 58.3 C
power avg 138.3 W peak 157.7 W
idle C2 avg 26.3% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
order-starved comm#328ab0c6 passover_picks 164 HIGH
held-cpu comm#94672850 held_ms 40.3017 HIGH
self-preempt tid=1649 stint_wake_ratio 9133 HIGH
self-preempt tid=1827 stint_wake_ratio 9129 HIGH
self-preempt tid=1650 stint_wake_ratio 9128 HIGH
self-preempt tid=1718 stint_wake_ratio 9128 HIGH
order-starved comm#328ab0c6 passover_picks 150 HIGH
held-cpu comm#c233d180 held_ms 26.2789 HIGH
self-preempt tid=1826 stint_wake_ratio 9128 HIGH
self-preempt tid=1652 stint_wake_ratio 9127 HIGH
self-preempt tid=1651 stint_wake_ratio 9124 HIGH
order-starved comm#328ab0c6 passover_picks 136 HIGH
held-cpu comm#8a5f93b2 held_ms 16.8421 HIGH
self-preempt tid=1717 stint_wake_ratio 9122 HIGH
kthread-strand comm#cfa26dc0 max_strand_ms 12.6346 MED
mean-reverting migration-rate hurst 0.474897 MED
discord 0ms profile_vs_mean 1.65273 MED
kthread-strand comm#ff850676 max_strand_ms 10.9932 MED
mean-reverting dispatch-rate hurst 0.356654 MED
hot-cpu 15 l2_miss_share 9.13602 LOW
CAPTURE complete -- 265373 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 2.7k wake2run; p50 937us p99 953us p999 957us worst 988us; 11.7% fast(<100us) / 22.1% mid / 66.2% tick-floor(>=900us); 44.4% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~2678 distinct values; inversion ratio 0.45
REPORT dispatch-stall
VERDICT: 1.8k saturated wakes over the 900us floor; PREEMPT-STARVED 41% (0 intervening picks) / ORDER-STARVED 59% (CPU served others first); avg 1.1 pass-overs, p99 2 pass-overs
REPORT kstrand
VERDICT: 3 strands across 2 per-CPU kthreads; worst HELD strand 12.6ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#cfa26dc0 15 2 12.6 12.6 2 0 comm#c233d180 99%
comm#ff850676 2 1 11.0 11.0 1 0 comm#f7d5fbc2 91%
WORKLOAD sojourn-pressure-16c-scx_flash
THERMAL/POWER
cpu temp peak 60.8 C avg 58.2 C
power avg 138.7 W peak 159.1 W
idle C2 avg 26.7% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
held-cpu comm#8a5f93b2 held_ms 41.1771 HIGH
self-preempt tid=1652 stint_wake_ratio 9128 HIGH
self-preempt tid=1651 stint_wake_ratio 9124 HIGH
self-preempt tid=1827 stint_wake_ratio 9124 HIGH
self-preempt tid=1826 stint_wake_ratio 9124 HIGH
held-cpu comm#8a5f93b2 held_ms 37.3917 HIGH
self-preempt tid=1649 stint_wake_ratio 9123 HIGH
self-preempt tid=1717 stint_wake_ratio 9122 HIGH
self-preempt tid=1650 stint_wake_ratio 9122 HIGH
held-cpu comm#8a5f93b2 held_ms 37.3083 HIGH
self-preempt tid=1718 stint_wake_ratio 9122 HIGH
order-starved comm#8a5f93b2 passover_picks 1 MED
mean-reverting dispatch-rate hurst 0.386384 MED
order-starved comm#8a5f93b2 passover_picks 1 MED
order-starved comm#8a5f93b2 passover_picks 1 MED
hot-cpu 15 l2_miss_share 8.59095 LOW
CAPTURE complete -- 346328 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 2.7k wake2run; p50 931us p99 946us p999 951us worst 957us; 14.9% fast(<100us) / 15.1% mid / 70.0% tick-floor(>=900us); 44.4% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~2588 distinct values; inversion ratio 0.62
REPORT dispatch-stall
VERDICT: 1.9k saturated wakes over the 900us floor; PREEMPT-STARVED 100% (0 intervening picks) / ORDER-STARVED 0% (CPU served others first); avg 2.0 pass-overs, p99 0 pass-overs
WORKLOAD sojourn-pressure-16c-scx_flow
THERMAL/POWER
cpu temp peak 60.8 C avg 57.8 C
power avg 133.6 W peak 158.9 W
idle C2 avg 36.5% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
self-preempt tid=1651 stint_wake_ratio 20043 HIGH
self-preempt tid=1827 stint_wake_ratio 19930 HIGH
self-preempt tid=1826 stint_wake_ratio 19887 HIGH
self-preempt tid=1652 stint_wake_ratio 19529 HIGH
self-preempt tid=1650 stint_wake_ratio 19322 HIGH
self-preempt tid=1718 stint_wake_ratio 19257 HIGH
self-preempt tid=1717 stint_wake_ratio 19251 HIGH
self-preempt tid=1649 stint_wake_ratio 19224 HIGH
discord 17258ms profile_vs_mean 1.75267 MED
hot-cpu 14 l2_miss_share 11.3479 MED
CAPTURE complete -- 582609 events, none dropped
KEY METRICS
REPORT sched
VERDICT: no WAKE2RUN events in trace (wake-to-run tracepoint not streamed?)
WORKLOAD sojourn-pressure-16c-scx_lavd
THERMAL/POWER
cpu temp peak 60.5 C avg 59.6 C
power avg 142.6 W peak 156.1 W
idle C2 avg 20.9% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
held-cpu comm#8a5f93b2 held_ms 62.1581 HIGH
self-preempt tid=1649 stint_wake_ratio 8454 HIGH
self-preempt tid=1652 stint_wake_ratio 8446 HIGH
self-preempt tid=1826 stint_wake_ratio 8435 HIGH
self-preempt tid=1718 stint_wake_ratio 8410 HIGH
held-cpu comm#8a5f93b2 held_ms 61.8241 HIGH
self-preempt tid=1717 stint_wake_ratio 8400 HIGH
self-preempt tid=1827 stint_wake_ratio 8399 HIGH
self-preempt tid=1651 stint_wake_ratio 8390 HIGH
held-cpu comm#8a5f93b2 held_ms 60.7793 HIGH
self-preempt tid=1650 stint_wake_ratio 8390 HIGH
order-starved comm#5a4f434f passover_picks 17 MED
kthread-strand comm#0878f080 max_strand_ms 14.009 MED
long-range-dependent migration-rate hurst 0.589445 MED
kthread-strand comm#180cd039 max_strand_ms 13.0008 MED
kthread-strand comm#12f62659 max_strand_ms 10.003 MED
kthread-strand comm#0357d927 max_strand_ms 9.99712 MED
kthread-strand comm#d4e6d4f1 max_strand_ms 9.99383 MED
kthread-strand comm#0dbd5298 max_strand_ms 9.97915 MED
order-starved comm#c233d180 passover_picks 12 MED
kthread-strand comm#ff850676 max_strand_ms 8.00513 MED
kthread-strand comm#fa4c3102 max_strand_ms 7.99979 MED
kthread-strand comm#cfa26f73 max_strand_ms 7.99631 MED
kthread-strand comm#493ffe92 max_strand_ms 6.9961 MED
kthread-strand comm#288f459f max_strand_ms 5.99881 MED
order-starved comm#328ab0c6 passover_picks 12 MED
kthread-strand comm#1beaeefc max_strand_ms 5.98868 MED
hot-cpu 15 l2_miss_share 9.27551 LOW
CAPTURE complete -- 227706 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 2.4k wake2run; p50 934us p99 2937us p999 3937us worst 6007us; 19.7% fast(<100us) / 19.3% mid / 61.1% tick-floor(>=900us); 9.9% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~2445 distinct values; inversion ratio 0.52
REPORT dispatch-stall
VERDICT: 1.5k saturated wakes over the 900us floor; PREEMPT-STARVED 92% (0 intervening picks) / ORDER-STARVED 8% (CPU served others first); avg 1.6 pass-overs, p99 2 pass-overs
REPORT kstrand
VERDICT: 51 strands across 12 per-CPU kthreads; worst HELD strand 14.0ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#0878f080 0 15 14.0 14.0 15 0 comm#8a5f93b2 50%
comm#180cd039 5 5 13.0 13.0 5 0 comm#8a5f93b2 38%
comm#12f62659 6 5 10.0 10.0 5 0 comm#8a5f93b2 30%
comm#0357d927 1 6 10.0 10.0 6 0 comm#8a5f93b2 60%
comm#d4e6d4f1 12 2 10.0 10.0 2 0 comm#8a5f93b2 100%
comm#0dbd5298 7 2 10.0 10.0 2 0 comm#8a5f93b2 60%
comm#ff850676 2 4 8.0 8.0 4 0 comm#8a5f93b2 63%
comm#fa4c3102 3 2 8.0 8.0 2 0 comm#8a5f93b2 37%
comm#cfa26f73 15 3 8.0 8.0 3 0 comm#c233d180 100%
comm#493ffe92 10 1 7.0 7.0 1 0 comm#8a5f93b2 100%
comm#288f459f 6 1 6.0 6.0 1 0 comm#94672850 50%
comm#1beaeefc 4 5 6.0 6.0 5 0 comm#8a5f93b2 50%
WORKLOAD sojourn-pressure-16c-scx_p2dq
THERMAL/POWER
cpu temp peak 60.8 C avg 55.0 C
power avg 127.6 W peak 160.6 W
idle C2 avg 45.3% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
order-starved comm#328ab0c6 passover_picks 20 HIGH
held-cpu comm#8a5f93b2 held_ms 16.3264 HIGH
self-preempt tid=1649 stint_wake_ratio 7022 HIGH
self-preempt tid=1650 stint_wake_ratio 7020 HIGH
self-preempt tid=1651 stint_wake_ratio 7012 HIGH
self-preempt tid=1827 stint_wake_ratio 6990 HIGH
order-starved comm#328ab0c6 passover_picks 9 HIGH
held-cpu comm#8a5f93b2 held_ms 4.94272 HIGH
self-preempt tid=1717 stint_wake_ratio 6988 HIGH
self-preempt tid=1718 stint_wake_ratio 5686 HIGH
self-preempt tid=1652 stint_wake_ratio 5594 HIGH
order-starved comm#8a5f93b2 passover_picks 8 HIGH
held-cpu comm#8a5f93b2 held_ms 4.86089 HIGH
self-preempt tid=1826 stint_wake_ratio 5575 HIGH
kthread-strand comm#0878f080 max_strand_ms 34.0008 MED
long-range-dependent migration-rate hurst 0.563128 MED
discord 9291ms profile_vs_mean 1.51849 MED
kthread-strand comm#1beaeefc max_strand_ms 24.8945 MED
kthread-strand comm#da1fa8b2 max_strand_ms 23.9963 MED
kthread-strand comm#e3146c3c max_strand_ms 22.9998 MED
kthread-strand comm#dca24885 max_strand_ms 22.9997 MED
kthread-strand comm#180cd039 max_strand_ms 21.9904 MED
kthread-strand comm#288f3ed3 max_strand_ms 20.9963 MED
kthread-strand comm#fa4c3102 max_strand_ms 20.9903 MED
kthread-strand comm#0dbd57b1 max_strand_ms 20.0006 MED
kthread-strand comm#ca8154b4 max_strand_ms 20.0004 MED
kthread-strand comm#cfa26dc0 max_strand_ms 20.0002 MED
kthread-strand comm#d4e6d4f1 max_strand_ms 19.9994 MED
kthread-strand comm#0357d927 max_strand_ms 19.9973 MED
kthread-strand comm#493ffe92 max_strand_ms 19.0002 MED
kthread-strand comm#12f62659 max_strand_ms 18.9978 MED
kthread-strand comm#e1c397f1 max_strand_ms 17.9989 MED
kthread-strand comm#ff850676 max_strand_ms 15.0007 MED
kthread-strand comm#180ccb20 max_strand_ms 10.9992 MED
hot-cpu 15 l2_miss_share 8.41489 LOW
CAPTURE complete -- 229302 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 1.9k wake2run; p50 2us p99 34935us p999 37958us worst 44934us; 90.9% fast(<100us) / 0.5% mid / 8.5% tick-floor(>=900us); 0.5% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~1244 distinct values; inversion ratio 0.63
REPORT dispatch-stall
VERDICT: 162 saturated wakes over the 900us floor; PREEMPT-STARVED 16% (0 intervening picks) / ORDER-STARVED 84% (CPU served others first); avg 5.0 pass-overs, p99 12 pass-overs
REPORT kstrand
VERDICT: 134 strands across 18 per-CPU kthreads; worst HELD strand 34.0ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#0878f080 0 32 34.0 34.0 32 0 comm#8a5f93b2 62%
comm#1beaeefc 4 10 24.9 24.9 10 0 comm#8a5f93b2 84%
comm#da1fa8b2 13 6 24.0 24.0 6 0 comm#8a5f93b2 88%
comm#e3146c3c 11 5 23.0 23.0 5 0 comm#8a5f93b2 91%
comm#dca24885 9 8 23.0 23.0 8 0 comm#8a5f93b2 91%
comm#180cd039 5 11 22.0 22.0 11 0 comm#8a5f93b2 95%
comm#288f3ed3 2 2 21.0 21.0 2 0 comm#8a5f93b2 100%
comm#fa4c3102 3 10 21.0 21.0 10 0 comm#8a5f93b2 95%
comm#0dbd57b1 7 5 20.0 20.0 5 0 comm#8a5f93b2 100%
comm#ca8154b4 14 2 20.0 20.0 2 0 comm#8a5f93b2 100%
comm#cfa26dc0 15 11 20.0 20.0 11 0 comm#c233d180 100%
comm#d4e6d4f1 12 3 20.0 20.0 3 0 comm#8a5f93b2 100%
comm#0357d927 1 8 20.0 20.0 8 0 comm#8a5f93b2 100%
comm#493ffe92 10 2 19.0 19.0 2 0 comm#8a5f93b2 100%
comm#12f62659 6 6 19.0 19.0 6 0 comm#8a5f93b2 100%
comm#e1c397f1 8 4 18.0 18.0 4 0 comm#8a5f93b2 100%
comm#ff850676 2 8 15.0 15.0 8 0 comm#8a5f93b2 100%
comm#180ccb20 5 1 11.0 11.0 1 0 comm#8a5f93b2 100%
WORKLOAD sojourn-pressure-16c-scx_rustland
THERMAL/POWER
cpu temp peak 61.0 C avg 58.7 C
power avg 132.7 W peak 157.2 W
idle C2 avg 33.6% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
order-starved comm#4fc5a1ab passover_picks 1599 HIGH
held-cpu comm#8a5f93b2 held_ms 2.77369 HIGH
self-preempt tid=275250 stint_wake_ratio 27525 HIGH
self-preempt tid=1652 stint_wake_ratio 6352 HIGH
self-preempt tid=1649 stint_wake_ratio 6351 HIGH
self-preempt tid=1718 stint_wake_ratio 6351 HIGH
order-starved comm#5a4f434f passover_picks 10 HIGH
self-preempt tid=1651 stint_wake_ratio 6350 HIGH
self-preempt tid=1827 stint_wake_ratio 6349 HIGH
self-preempt tid=1717 stint_wake_ratio 6349 HIGH
order-starved comm#328ab0c6 passover_picks 2 HIGH
self-preempt tid=1826 stint_wake_ratio 6347 HIGH
kthread-strand comm#0878f080 max_strand_ms 22.0024 MED
long-range-dependent migration-rate hurst 0.610515 MED
discord 4342ms profile_vs_mean 1.52759 MED
kthread-strand comm#ff850676 max_strand_ms 20.9967 MED
kthread-strand comm#12f62659 max_strand_ms 20.0078 MED
kthread-strand comm#fa4c3102 max_strand_ms 19.9998 MED
kthread-strand comm#0357d927 max_strand_ms 16.0016 MED
kthread-strand comm#1beaeefc max_strand_ms 13.0003 MED
kthread-strand comm#cfa26dc0 max_strand_ms 10.9973 MED
kthread-strand comm#d4e6d4f1 max_strand_ms 7.99943 MED
kthread-strand comm#0dbd57b1 max_strand_ms 5.99039 MED
hot-cpu 7 l2_miss_share 8.2473 LOW
CAPTURE complete -- 269609 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 2.6k wake2run; p50 2us p99 32512us p999 41958us worst 42998us; 91.3% fast(<100us) / 1.4% mid / 7.2% tick-floor(>=900us); 6.1% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~1956 distinct values; inversion ratio 0.55
REPORT dispatch-stall
VERDICT: 189 saturated wakes over the 900us floor; PREEMPT-STARVED 1% (0 intervening picks) / ORDER-STARVED 99% (CPU served others first); avg 17.4 pass-overs, p99 41 pass-overs
REPORT kstrand
VERDICT: 17 strands across 9 per-CPU kthreads; worst HELD strand 22.0ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#0878f080 0 5 22.0 22.0 5 0 comm#8a5f93b2 100%
comm#ff850676 2 1 21.0 21.0 1 0 comm#8a5f93b2 100%
comm#12f62659 6 3 20.0 20.0 3 0 comm#8a5f93b2 100%
comm#fa4c3102 3 1 20.0 20.0 1 0 comm#8a5f93b2 100%
comm#0357d927 1 1 16.0 16.0 1 0 comm#8a5f93b2 100%
comm#1beaeefc 4 1 13.0 13.0 1 0 comm#8a5f93b2 100%
comm#cfa26dc0 15 3 11.0 11.0 3 0 comm#c233d180 100%
comm#d4e6d4f1 12 1 8.0 8.0 1 0 comm#8a5f93b2 100%
comm#0dbd57b1 7 1 6.0 6.0 1 0 comm#8a5f93b2 99%
WORKLOAD sojourn-pressure-16c-scx_rusty
THERMAL/POWER
cpu temp peak 62.0 C avg 56.2 C
power avg 124.1 W peak 155.6 W
idle C2 avg 45.5% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
order-starved comm#328ab0c6 passover_picks 450 HIGH
held-cpu comm#8a5f93b2 held_ms 6.94375 HIGH
self-preempt tid=1718 stint_wake_ratio 8104 HIGH
self-preempt tid=1827 stint_wake_ratio 8088 HIGH
self-preempt tid=1649 stint_wake_ratio 8066 HIGH
self-preempt tid=1651 stint_wake_ratio 8051 HIGH
order-starved comm#328ab0c6 passover_picks 207 HIGH
held-cpu comm#8a5f93b2 held_ms 6.87148 HIGH
self-preempt tid=1717 stint_wake_ratio 7824 HIGH
self-preempt tid=1650 stint_wake_ratio 7820 HIGH
self-preempt tid=1826 stint_wake_ratio 7802 HIGH
order-starved comm#328ab0c6 passover_picks 96 HIGH
held-cpu comm#8a5f93b2 held_ms 5.80406 HIGH
self-preempt tid=1652 stint_wake_ratio 7493 HIGH
kthread-strand comm#0357d927 max_strand_ms 57.0009 MED
long-range-dependent dispatch-rate hurst 0.556989 MED
kthread-strand comm#da1fa8b2 max_strand_ms 38.0053 MED
kthread-strand comm#0878f080 max_strand_ms 30.0052 MED
kthread-strand comm#dca24885 max_strand_ms 25.9998 MED
kthread-strand comm#cfa26f73 max_strand_ms 25.0047 MED
kthread-strand comm#e1c397f1 max_strand_ms 24.9992 MED
kthread-strand comm#493ffe92 max_strand_ms 23.0087 MED
kthread-strand comm#ff850676 max_strand_ms 23.0003 MED
kthread-strand comm#ca8154b4 max_strand_ms 22.999 MED
kthread-strand comm#0dbd5298 max_strand_ms 21.9991 MED
kthread-strand comm#d4e6d4f1 max_strand_ms 20.9997 MED
kthread-strand comm#2f099b5a max_strand_ms 20.004 MED
kthread-strand comm#12f62659 max_strand_ms 20.0011 MED
kthread-strand comm#180cd039 max_strand_ms 20.0009 MED
kthread-strand comm#fa4c3102 max_strand_ms 19.9981 MED
kthread-strand comm#e3146c3c max_strand_ms 19.9978 MED
kthread-strand comm#658ebbd8 max_strand_ms 19.0003 MED
kthread-strand comm#1beaeefc max_strand_ms 19.0001 MED
kthread-strand comm#ca81581a max_strand_ms 15.9992 MED
kthread-strand comm#cfa272d9 max_strand_ms 15.9946 MED
kthread-strand comm#da1faa65 max_strand_ms 15.9062 MED
kthread-strand comm#0dbd55fe max_strand_ms 15 MED
kthread-strand comm#0dbd57b1 max_strand_ms 14.9661 MED
kthread-strand comm#eb6af1dc max_strand_ms 12.9986 MED
kthread-strand comm#dca2451f max_strand_ms 10.0046 MED
kthread-strand comm#cfa26dc0 max_strand_ms 9.00624 MED
hot-cpu 3 l2_miss_share 7.45899 LOW
CAPTURE complete -- 206519 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 2.1k wake2run; p50 877us p99 9023us p999 12868us worst 16201us; 45.8% fast(<100us) / 9.0% mid / 45.2% tick-floor(>=900us); 0.1% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~2141 distinct values; inversion ratio 0.54
REPORT dispatch-stall
VERDICT: 968 saturated wakes over the 900us floor; PREEMPT-STARVED 6% (0 intervening picks) / ORDER-STARVED 94% (CPU served others first); avg 3.7 pass-overs, p99 9 pass-overs
REPORT kstrand
VERDICT: 288 strands across 26 per-CPU kthreads; worst HELD strand 57.0ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#0357d927 1 31 57.0 57.0 31 0 comm#8a5f93b2 63%
comm#da1fa8b2 13 24 38.0 38.0 24 0 comm#8a5f93b2 55%
comm#0878f080 0 29 30.0 30.0 29 0 comm#8a5f93b2 100%
comm#dca24885 9 15 26.0 26.0 15 0 comm#8a5f93b2 100%
comm#cfa26f73 15 18 25.0 25.0 18 0 comm#8a5f93b2 100%
comm#e1c397f1 8 7 25.0 25.0 7 0 comm#8a5f93b2 84%
comm#493ffe92 10 11 23.0 23.0 11 0 comm#8a5f93b2 100%
comm#ff850676 2 11 23.0 23.0 11 0 comm#8a5f93b2 100%
comm#ca8154b4 14 31 23.0 23.0 31 0 comm#8a5f93b2 91%
comm#0dbd5298 7 12 22.0 22.0 12 0 comm#8a5f93b2 100%
comm#d4e6d4f1 12 15 21.0 21.0 15 0 comm#8a5f93b2 100%
comm#2f099b5a 2 1 20.0 20.0 1 0 comm#8a5f93b2 100%
comm#12f62659 6 13 20.0 20.0 13 0 comm#8a5f93b2 100%
comm#180cd039 5 17 20.0 20.0 17 0 comm#8a5f93b2 100%
comm#fa4c3102 3 10 20.0 20.0 10 0 comm#8a5f93b2 100%
comm#e3146c3c 11 13 20.0 20.0 13 0 comm#8a5f93b2 100%
comm#658ebbd8 0 10 19.0 19.0 10 0 comm#8a5f93b2 100%
comm#1beaeefc 4 9 19.0 19.0 9 0 comm#8a5f93b2 100%
comm#ca81581a 14 3 16.0 16.0 3 0 comm#8a5f93b2 100%
comm#cfa272d9 15 2 16.0 16.0 2 0 comm#c233d180 100%
WORKLOAD sojourn-pressure-16c-scx_tickless
THERMAL/POWER
cpu temp peak 61.0 C avg 58.1 C
power avg 139.5 W peak 157.3 W
idle C2 avg 24.1% (dominant)
POORLY-BEHAVING ITEMS (ranked)
kind id metric value sev
order-starved comm#328ab0c6 passover_picks 169 HIGH
held-cpu comm#8a5f93b2 held_ms 11.3836 HIGH
self-preempt tid=1650 stint_wake_ratio 6470 HIGH
self-preempt tid=1651 stint_wake_ratio 6466 HIGH
self-preempt tid=1826 stint_wake_ratio 6466 HIGH
self-preempt tid=1652 stint_wake_ratio 6466 HIGH
order-starved comm#328ab0c6 passover_picks 157 HIGH
held-cpu comm#8a5f93b2 held_ms 6.87826 HIGH
self-preempt tid=1649 stint_wake_ratio 6465 HIGH
self-preempt tid=1827 stint_wake_ratio 6465 HIGH
self-preempt tid=1717 stint_wake_ratio 6465 HIGH
order-starved comm#328ab0c6 passover_picks 151 HIGH
held-cpu comm#8a5f93b2 held_ms 3.94353 HIGH
self-preempt tid=1718 stint_wake_ratio 6454 HIGH
kthread-strand comm#493ffe92 max_strand_ms 40.9941 MED
discord 3816ms profile_vs_mean 1.86104 MED
hot-cpu 15 l2_miss_share 10.2089 MED
kthread-strand comm#fa4c3102 max_strand_ms 32.9976 MED
kthread-strand comm#0dbd57b1 max_strand_ms 29.0008 MED
kthread-strand comm#d4e6d4f1 max_strand_ms 25.9973 MED
kthread-strand comm#ca8154b4 max_strand_ms 24.0003 MED
kthread-strand comm#0878f080 max_strand_ms 21.0001 MED
kthread-strand comm#e3146c3c max_strand_ms 20.9914 MED
kthread-strand comm#dca24885 max_strand_ms 20.8888 MED
kthread-strand comm#e1c397f1 max_strand_ms 20.0134 MED
kthread-strand comm#1beaeefc max_strand_ms 19.9976 MED
kthread-strand comm#da1fa8b2 max_strand_ms 19.9959 MED
kthread-strand comm#ff850676 max_strand_ms 19.9944 MED
kthread-strand comm#0357d927 max_strand_ms 19.9935 MED
kthread-strand comm#cfa26f73 max_strand_ms 19.9921 MED
kthread-strand comm#2f099b5a max_strand_ms 19.0055 MED
kthread-strand comm#180cd039 max_strand_ms 17.9992 MED
kthread-strand comm#12f62659 max_strand_ms 17.993 MED
kthread-strand comm#288f4905 max_strand_ms 15.9494 MED
kthread-strand comm#ddbfab61 max_strand_ms 12.9981 MED
kthread-strand comm#cfa26dc0 max_strand_ms 11.157 MED
CAPTURE complete -- 153433 events, none dropped
KEY METRICS
REPORT sched
VERDICT: 1.4k wake2run; p50 1945us p99 9489us p999 11008us worst 22939us; 10.3% fast(<100us) / 11.3% mid / 78.4% tick-floor(>=900us); 47.7% cross-domain
STRUCTURE: latency-over-trace RANDOM; ~1393 distinct values; inversion ratio 0.63
REPORT dispatch-stall
VERDICT: 1.1k saturated wakes over the 900us floor; PREEMPT-STARVED 1% (0 intervening picks) / ORDER-STARVED 99% (CPU served others first); avg 6.0 pass-overs, p99 14 pass-overs
REPORT kstrand
VERDICT: 254 strands across 20 per-CPU kthreads; worst HELD strand 41.0ms (I/O-completion freeze signature)
kthread cpu strands max_ms p99_ms held dark held_by
comm#493ffe92 10 8 41.0 41.0 8 0 comm#8a5f93b2 51%
comm#fa4c3102 3 12 33.0 33.0 12 0 comm#8a5f93b2 64%
comm#0dbd57b1 7 15 29.0 29.0 15 0 comm#94672850 72%
comm#d4e6d4f1 12 10 26.0 26.0 10 0 comm#8a5f93b2 81%
comm#ca8154b4 14 15 24.0 24.0 15 0 comm#8a5f93b2 87%
comm#0878f080 0 44 21.0 21.0 44 0 comm#94672850 100%
comm#e3146c3c 11 10 21.0 21.0 10 0 comm#8a5f93b2 100%
comm#dca24885 9 9 20.9 20.9 9 0 comm#8a5f93b2 100%
comm#e1c397f1 8 32 20.0 20.0 32 0 comm#8a5f93b2 100%
comm#1beaeefc 4 22 20.0 20.0 22 0 comm#8a5f93b2 100%
comm#da1fa8b2 13 10 20.0 20.0 10 0 comm#8a5f93b2 100%
comm#ff850676 2 16 20.0 20.0 16 0 comm#8a5f93b2 100%
comm#0357d927 1 13 20.0 20.0 13 0 comm#8a5f93b2 100%
comm#cfa26f73 15 12 20.0 20.0 12 0 comm#8a5f93b2 100%
comm#2f099b5a 2 3 19.0 19.0 3 0 comm#8a5f93b2 100%
comm#180cd039 5 9 18.0 18.0 9 0 comm#8a5f93b2 100%
comm#12f62659 6 11 18.0 18.0 11 0 comm#8a5f93b2 100%
comm#288f4905 4 1 15.9 15.9 1 0 comm#8a5f93b2 100%
comm#ddbfab61 5 1 13.0 13.0 1 0 comm#8a5f93b2 100%
comm#cfa26dc0 15 1 11.2 11.2 1 0 comm#c233d180 0%
