Oblivious Algorithms Benchmarks

This repository contains reproducible benchmarks related to several implementations of ORAM, Oblivious Maps and other oblivious algorithms targeting TEEs (Trusted Execution Environments). To reproduce, simply run sh go.sh. Our goal with this repo is to provide transparent analysis of the performance of several oblivious algorithms, and to incentivize the development of efficient implementations of oblivious algorithms.

See benchmark types for information about the algorithms that are being benchmarked and implementations for each algorithm and benchmark results for a summary of the results of each implementation. Contributions with new implementations are welcome, see CONTRIBUTING.md for information on how to add new benchmarks.

Benchmark Types

Name	Description	Measured Parameters	Independent Variables
NRORAM	Non recursive ORAM (Requires storing the position somewhere for accesses)	Read Latency, Read Throughput, Initialization Time, Memory Usage	N (ORAM size), K (Address size), V (Value size)
RORAM	Recursive ORAM - Basically an array	Read Latency, Read Throughput, Initialization Time, Memory Usage	N (ORAM size), K (Address size), V (Value size)
UMAP	Unordered Map / Key-Value Store (Many disperse keys)	Read Latency, Read Throughput, Initialization Time, Memory Usage	N (ORAM size), K (Key size), V (Value size)
Sharded-UMAP	Sharded UMap	Batch Read Latency, Batch Read Throughput, Initialization Time, Memory Usage	N (ORAM size), K (Key size), V (Value size), P (Number of Partitions), B (Queries per batch)
Ramp-UMAP	Open-loop point lookups with ramp, constant, or intermittent input	End-to-end average and tail latency, raw arrival/response timestamps, dynamic caller batch sizes	N (map size), workload and input-rate controls

Benchmarked Implementations

Name	Description	Language	Types
olabs_oram	Oblivious Labs's SGX implementation of ORAM and UMAP	C++	NRORAM, RORAM, UMAP, Sharded-UMAP, Ramp-UMAP
signal_icelake	Signal's pre-Jasmin implementation of ORAM and UMAP for Private Contact Discovery	C++	UMAP, Sharded-UMAP, Ramp-UMAP
signal_jasmine	Signal's Jasmin-backed implementation of ORAM and UMAP for Private Contact Discovery	C/Jasmin	UMAP, Sharded-UMAP, Ramp-UMAP
olabs_rostl	Oblivious Labs's Rust Oblivious Standard Library implementation of ORAM and UMAP	Rust	NRORAM, RORAM, UMAP, Sharded-UMAP, Ramp-UMAP
mc_oblivious	Mobilecoin's implementation of ORAM and UMAP	Rust	RORAM, UMAP, Ramp-UMAP
meta_oram	Meta's implementation of RORAM	Rust	RORAM
h2o2_oram	H2O2RAM's high-performance hierarchical doubly oblivious RAM	C++	RORAM, Ramp-UMAP
sonic	Native SONIC RingORAM and PMCHAIN batched OMAP (details)	C++	NRORAM, Batched-UMAP, Ramp-UMAP
cpp_stl	Non-oblivious baseline: contiguous arrays (std::vector) and std::unordered_map; N insertions and N shuffled successful queries	C++	RORAM baseline, UMAP baseline

Benchmark Results

Environment

All the benchmarks are ran on the same environment. A mini server with an AMD Ryzen 9 7945HX with 64GB of RAM, and an WD Black SN850 SSD, running an up to date version of arch linux.

Non-Recursive ORAM (NRORAM)

8B keys, 8B values

Read latency (us/query)

N	olabs_rostl	olabs_oram	sonic_ser	sonic_par8	sonic_par16
$2^{10}$	0.52	0.50	1.17	7.65	15.39
$2^{11}$	0.55	0.52	1.25	8.06	16.70
$2^{12}$	0.62	0.55	1.38	8.01	17.62
$2^{13}$	0.62	0.58	1.49	9.15	18.77
$2^{14}$	0.66	0.61	1.59	9.87	20.11
$2^{15}$	0.71	0.64	1.69	9.78	20.94
$2^{16}$	0.76	0.68	1.79	10.32	21.82
$2^{17}$	0.82	0.72	1.93	10.63	23.04
$2^{18}$	0.85	0.76	2.28	11.67	24.08
$2^{19}$	0.96	0.79	2.61	11.93	25.68
$2^{20}$	1.04	0.83	2.98	12.70	26.81
$2^{21}$	1.11	0.84	3.26	13.53	28.34
$2^{22}$	1.19	0.88	3.61	13.83	29.68
$2^{23}$	1.25	0.91	3.91	14.86	31.07
$2^{24}$	1.38	0.98	4.22	15.16	32.21
$2^{25}$	1.56	0.99	4.51	16.16	33.67
$2^{26}$	1.75	1.06	4.90	16.50	34.89
$2^{27}$	1.95	1.15	5.07	16.82	36.21
$2^{28}$	2.23	1.23		18.20	38.43

Read throughput (queries/s)

N	olabs_rostl	olabs_oram	sonic_ser	sonic_par8	sonic_par16
$2^{10}$	1928825.16	2016869.54	842404.69	1000677.11	890577.72
$2^{11}$	1816511.88	1909309.48	787269.81	950217.38	839179.80
$2^{12}$	1612591.72	1803035.78	716726.73	928340.12	797485.95
$2^{13}$	1610848.59	1719096.38	664296.57	854668.79	751205.25
$2^{14}$	1508030.43	1629424.57	621270.71	804820.36	721036.63
$2^{15}$	1411520.19	1554133.27	584930.32	789443.46	693499.95
$2^{16}$	1318502.27	1469738.03	551910.85	748507.13	669432.03
$2^{17}$	1219351.72	1397981.18	512039.83	721647.84	641886.21
$2^{18}$	1171158.50	1312881.27	433939.74	680260.47	620924.86
$2^{19}$	1040668.70	1266553.46	380486.91	641392.55	580137.35
$2^{20}$	958758.92	1203820.34	333788.70	623927.10	553274.72
$2^{21}$	900164.49	1184733.50	305380.94	586902.62	527816.64
$2^{22}$	843527.27	1130075.98	275486.49	562109.28	501274.94
$2^{23}$	800475.04	1104972.05	254610.36	534526.85	481012.92
$2^{24}$	724106.75	1022250.85	236073.41	515531.40	464342.39
$2^{25}$	641576.16	1009048.70	220959.53	491274.44	443025.62
$2^{26}$	571700.15	946882.26	203294.80	474445.97	427821.07
$2^{27}$	511855.26	872830.93	196520.79	464817.02	412054.26
$2^{28}$	447600.72	809816.74		430923.52	384577.89

Recursive ORAM (RORAM)

8B keys, 8B values

Read latency (us/query)

N	Signal-Old	Signal-Jasmine	olabs_rostl	olabs_oram	meta_oram
$2^{10}$	2.37		0.59	0.17	32.27
$2^{11}$	3.10		0.68	0.28	36.73
$2^{12}$	3.37		1.38	0.78	42.89
$2^{13}$	4.09		1.38	0.86	51.96
$2^{14}$	4.38		1.50	0.96	67.10
$2^{15}$	5.50		1.47	1.12	92.76
$2^{16}$	6.16		2.59	1.60	141.60
$2^{17}$	7.29	25.12	2.45	1.73	234.92
$2^{18}$	8.20	28.70	2.64	1.88	151.74
$2^{19}$	9.55	31.27	2.67	2.14	204.57
$2^{20}$	11.70	34.28	4.43	2.58	303.43
$2^{21}$	15.24	36.90	4.36	2.82	221.66
$2^{22}$	16.91	40.14	4.41	3.16	280.35
$2^{23}$	18.99	43.67	4.43	3.44	385.43
$2^{24}$	20.74	47.03	6.62	4.07	311.06
$2^{25}$	23.22	52.95	6.70	4.55	370.09
$2^{26}$	26.36	56.62	7.03	4.66	475.65
$2^{27}$	29.95	62.37	7.18	5.08	412.60
$2^{28}$	35.71	68.46		5.87	498.07

8B keys, 56B values

Read latency (us/query)

N	Signal-Old	Signal-Jasmine	olabs_oram	meta_oram	h2o2*
$2^{10}$	4.22		0.77	28.12	3.85
$2^{11}$	4.46		0.88	32.12	4.04
$2^{12}$	5.49		1.12	38.06	3.87
$2^{13}$	6.12		1.54	47.28	3.73
$2^{14}$	7.36	25.33	1.66	61.55	3.77
$2^{15}$	8.11	28.23	1.87	87.78	3.87
$2^{16}$	9.41	31.36	2.09	136.21	3.87
$2^{17}$	11.44	34.20	2.54	229.95	3.88
$2^{18}$	14.77	37.58	2.75	145.11	4.88
$2^{19}$	16.72	40.11	3.02	198.18	6.19
$2^{20}$	19.10	43.48	3.43	296.32	7.09
$2^{21}$	20.89	46.90	4.00	218.11	8.32
$2^{22}$	23.32	52.49	4.35	273.28	9.40
$2^{23}$	26.37	56.64	4.81	379.06	9.94
$2^{24}$	29.65	62.25	5.42	303.79	11.60
$2^{25}$	34.89	68.04	6.29	367.62	12.98
$2^{26}$	37.40	81.15	6.80	471.52	14.59
$2^{27}$	40.50	94.84	7.46	409.08
$2^{28}$			8.08

*-average latency

Unordered Map - Single Queries

8B keys, 56B values

Read latency (us/query)

N	Signal-Old	Signal-Jasmine	olabs_rostl	olabs_umap
$2^{10}$	7.80		2.09	1.87
$2^{11}$	8.46		2.21	3.15
$2^{12}$	10.79		2.57	3.00
$2^{13}$	12.38		4.18	3.16
$2^{14}$	14.66	51.22	4.39	3.48
$2^{15}$	16.07	57.91	4.52	4.19
$2^{16}$	19.65	62.81	4.85	4.84
$2^{17}$	24.70	68.52	7.78	5.42
$2^{18}$	33.54		8.19	6.04
$2^{19}$	35.01	81.12	8.57	6.87
$2^{20}$	39.20	88.11	9.03	7.96
$2^{21}$	44.29	96.33	12.70	8.59
$2^{22}$	50.16	107.55	13.62	9.63
$2^{23}$	57.45	119.39	13.56	10.18
$2^{24}$	67.47	135.75	14.20	11.45
$2^{25}$	82.39		18.85	12.66
$2^{26}$			19.71	13.66
$2^{27}$				14.51

8B keys, 8B values

Read latency (us/query)

N	mc_oblivious	olabs_rostl
$2^{10}$	83.87	1.18
$2^{11}$	102.08	1.30
$2^{12}$	124.32	1.50
$2^{13}$	147.04	3.01
$2^{14}$	171.63	3.14
$2^{15}$	198.71	3.23
$2^{16}$	229.66	3.37
$2^{17}$	313.53	5.70
$2^{18}$	360.50	6.02
$2^{19}$	411.84	6.16
$2^{20}$	464.85	6.47
$2^{21}$	521.01	9.78
$2^{22}$	583.35	10.08
$2^{23}$	645.46	10.10
$2^{24}$	715.51	10.36
$2^{25}$	842.09	14.69
$2^{26}$	927.84	15.08
$2^{27}$	1012.94	15.67
$2^{28}$	1132.16	16.81

Unordered Map - Batched Queries

Each comparison fixes the batch size.

8B keys, 56B values, 1K queries/batch

Batch read latency (us/batch)

N	olabs_rostl_sharded (S=15)	olabs_umap_sharded (S=15)	sonic_pmchain
$2^{10}$	509.27	725.21
$2^{11}$	554.41	707.97	1096.72
$2^{12}$	586.52	786.02	1309.22
$2^{13}$	618.55	828.68	1612.86
$2^{14}$	655.05	1183.72	2107.44
$2^{15}$	671.08	1263.35	3112.89
$2^{16}$	718.20	1350.22	4919.82
$2^{17}$	1098.14	1426.74	8548.79
$2^{18}$	1178.84	1809.74	15585.89
$2^{19}$	1350.41	1933.15	29509.56
$2^{20}$	1651.10	2072.44	57304.28
$2^{21}$	2613.50	2462.02	112508.16
$2^{22}$	2940.12	2941.73	217436.17
$2^{23}$	3352.56	3228.46	431636.06
$2^{24}$	3795.45	3611.53	883498.55
$2^{25}$	8029.02	4213.20
$2^{26}$	9528.76	4369.93
$2^{27}$		5064.53

Read throughput (queries/s)

N	olabs_rostl_sharded (S=15)	olabs_umap_sharded (S=15)	sonic_pmchain
$2^{10}$	2010734.31	1411996.10
$2^{11}$	1847019.30	1446385.74	921282.96
$2^{12}$	1745901.54	1302762.37	773445.95
$2^{13}$	1655483.63	1235707.44	629110.75
$2^{14}$	1563243.57	865072.70	482549.41
$2^{15}$	1525895.69	810542.00	327414.20
$2^{16}$	1425790.34	758396.07	207518.83
$2^{17}$	932489.83	717721.88	119578.65
$2^{18}$	868653.44	565828.32	65638.76
$2^{19}$	758286.80	529704.92	34683.11
$2^{20}$	620194.35	494103.78	17864.84
$2^{21}$	391811.58	415919.05	9100.34
$2^{22}$	348285.35	348094.62	4709.05
$2^{23}$	305438.58	317179.57	2372.26
$2^{24}$	269796.73	283536.54	1159.01
$2^{25}$	127537.36	243045.92
$2^{26}$	107464.11	234328.88
$2^{27}$		202190.35

8B keys, 56B values, 4K queries/batch

Batch read latency (us/batch)

N	olabs_rostl_sharded (S=15)	olabs_umap_sharded (S=15)	sonic_pmchain
$2^{10}$	1820.08	2716.80
$2^{11}$	1922.97	2587.60
$2^{12}$	2024.10	2559.01	3016.52
$2^{13}$	2139.99	2628.93	3359.04
$2^{14}$	2203.07	4100.40	3974.79
$2^{15}$	2288.13	4066.32	5119.80
$2^{16}$	2578.84	4407.14	7149.49
$2^{17}$	3863.84	4323.60	10933.58
$2^{18}$	4295.65	5627.45	18132.26
$2^{19}$	4751.03	5840.88	32157.30
$2^{20}$	5776.41	6440.66	59771.35
$2^{21}$	9207.57	7250.93	114406.63
$2^{22}$	10856.96	8210.50	221114.28
$2^{23}$	12286.73	9410.63	434884.66
$2^{24}$	13916.43	10303.10	869646.90
$2^{25}$	30537.95	12129.74
$2^{26}$	35631.54	12562.40
$2^{27}$		14478.81

Read throughput (queries/s)

N	olabs_rostl_sharded (S=15)	olabs_umap_sharded (S=15)	sonic_pmchain
$2^{10}$	2250453.00	1507653.39
$2^{11}$	2130040.45	1582934.00
$2^{12}$	2023620.38	1600621.99	1331398.38
$2^{13}$	1914030.35	1558046.14	1198351.36
$2^{14}$	1859225.57	998927.58	1015442.16
$2^{15}$	1790109.54	1007298.61	790934.41
$2^{16}$	1588311.32	929401.12	568200.79
$2^{17}$	1060086.27	947357.93	372601.88
$2^{18}$	953521.81	727861.05	225165.28
$2^{19}$	862129.40	701263.74	127139.76
$2^{20}$	709090.34	635959.47	68460.39
$2^{21}$	444851.50	564893.06	35783.67
$2^{22}$	377269.66	498873.62	18519.38
$2^{23}$	333367.74	435252.27	9417.21
$2^{24}$	294328.46	397550.22	4709.63
$2^{25}$	134128.18	337682.40
$2^{26}$	114954.33	326052.28
$2^{27}$		282896.19

8B keys, 56B values, 64K queries/batch

Batch read latency (us/batch)

N	Signal-Old-Sharded (S=15)	Signal-Jasmine-Sharded (S=16)	olabs_rostl_sharded (S=15)	olabs_umap_sharded (S=15)	sonic_pmchain
$2^{10}$			36196.54	40467.92
$2^{11}$			35890.80	41094.14
$2^{12}$			37596.60	40868.15
$2^{13}$			37761.02	39296.03
$2^{14}$			39618.54	50563.96
$2^{15}$			42739.19	51435.96
$2^{16}$	116052.87		43437.00	52482.03	63318.50
$2^{17}$	121759.01		55306.49	53161.48	72756.74
$2^{18}$	133822.46	753681.70	60481.32	65453.83	84910.35
$2^{19}$	148504.04	478991.49	72127.46	70281.81	105403.29
$2^{20}$	183116.78	1034319.85	85185.49	75747.45	139059.29
$2^{21}$	224846.28	683314.80	130970.28	82533.79	200126.05
$2^{22}$	219103.24	733329.43	149260.25	92448.36	312129.84
$2^{23}$	314380.18	781689.72	169341.27	107997.37	529793.90
$2^{24}$	283359.49	1592207.61	195399.80	116054.78	967443.37
$2^{25}$	440473.38	1118394.64	430856.13	134284.73
$2^{26}$	617673.94	987875.12	507015.31	140207.00
$2^{27}$	704199.27	1571824.77		160596.51

Read throughput (queries/s)

N	Signal-Old-Sharded (S=15)	Signal-Jasmine-Sharded (S=16)	olabs_rostl_sharded (S=15)	olabs_umap_sharded (S=15)	sonic_pmchain
$2^{10}$			1810559.69	1619455.42
$2^{11}$			1825983.26	1594777.33
$2^{12}$			1743136.44	1603595.76
$2^{13}$			1735546.44	1667751.21
$2^{14}$			1654174.93	1296101.08
$2^{15}$			1533393.69	1274128.04
$2^{16}$	564708.12		1508759.83	1248732.22	1019024.18
$2^{17}$	538243.55		1184960.30	1232772.36	888694.10
$2^{18}$	489723.46	86954.48	1083574.15	1001255.42	762964.26
$2^{19}$	441307.85	136820.80	908613.73	932474.60	616003.37
$2^{20}$	357891.84	63361.44	769332.85	865190.89	467957.75
$2^{21}$	291470.25	95908.94	500388.35	794050.54	325855.03
$2^{22}$	299110.13	89367.75	439072.02	708893.04	209302.84
$2^{23}$	208460.98	83838.89	387005.49	606829.58	123472.50
$2^{24}$	231282.18	41160.46	335394.41	564698.86	67672.98
$2^{25}$	148785.38	58598.28	152106.46	488037.64
$2^{26}$	106101.29	66340.37	129258.42	467423.16
$2^{27}$	93064.57	41694.22		408078.62

Unordered Map - Scaling with Batch Size

olabs_umap_sharded (S=15): 8B keys, 56B values

Read throughput (queries/s), by batch size

N	B=1K	B=4K	B=8K	B=64K	B=1M
$2^{10}$	1411996.10	1507653.39	1469382.07	1619455.42	1276371.51
$2^{11}$	1446385.74	1582934.00	1487821.20	1594777.33	1259970.56
$2^{12}$	1302762.37	1600621.99	1404509.76	1603595.76	1275598.92
$2^{13}$	1235707.44	1558046.14	1458917.69	1667751.21	1260924.42
$2^{14}$	865072.70	998927.58	1030260.18	1296101.08	1162661.50
$2^{15}$	810542.00	1007298.61	999695.40	1274128.04	1142580.39
$2^{16}$	758396.07	929401.12	1002928.30	1248732.22	1148382.21
$2^{17}$	717721.88	947357.93	952547.32	1232772.36	1117377.33
$2^{18}$	565828.32	727861.05	770461.69	1001255.42	964627.70
$2^{19}$	529704.92	701263.74	722838.75	932474.60	913578.81
$2^{20}$	494103.78	635959.47	667557.04	865190.89	880696.20
$2^{21}$	415919.05	564893.06	587492.68	794050.54	817829.52
$2^{22}$	348094.62	498873.62	525098.48	708893.04	749618.74
$2^{23}$	317179.57	435252.27	452877.52	606829.58	648841.50
$2^{24}$	283536.54	397550.22	407311.86	564698.86	607009.52
$2^{25}$	243045.92	337682.40	353673.64	488037.64
$2^{26}$	234328.88	326052.28	338621.19	467423.16	523110.65
$2^{27}$	202190.35	282896.19	299804.73	408078.62

Batch read latency (us/batch), by batch size

N	B=1K	B=4K	B=8K	B=64K	B=1M
$2^{10}$	725.21	2716.80	5575.13	40467.92	821528.84
$2^{11}$	707.97	2587.60	5506.04	41094.14	832222.62
$2^{12}$	786.02	2559.01	5832.64	40868.15	822026.41
$2^{13}$	828.68	2628.93	5615.12	39296.03	831593.06
$2^{14}$	1183.72	4100.40	7951.39	50563.96	901875.57
$2^{15}$	1263.35	4066.32	8194.50	51435.96	917726.24
$2^{16}$	1350.22	4407.14	8168.08	52482.03	913089.73
$2^{17}$	1426.74	4323.60	8600.10	53161.48	938426.06
$2^{18}$	1809.74	5627.45	10632.59	65453.83	1087026.63
$2^{19}$	1933.15	5840.88	11333.09	70281.81	1147767.42
$2^{20}$	2072.44	6440.66	12271.61	75747.45	1190621.69
$2^{21}$	2462.02	7250.93	13944.00	82533.79	1282144.97
$2^{22}$	2941.73	8210.50	15600.88	92448.36	1398812.42
$2^{23}$	3228.46	9410.63	18088.78	107997.37	1616074.19
$2^{24}$	3611.53	10303.10	20112.35	116054.78	1727445.72
$2^{25}$	4213.20	12129.74	23162.60	134284.73
$2^{26}$	4369.93	12562.40	24192.23	140207.00	2004501.33
$2^{27}$	5064.53	14478.81	27324.45	160596.51

sonic_pmchain: 8B keys, 56B values

Read throughput (queries/s), by batch size

N	B=1K	B=4K	B=8K	B=64K	B=1M
$2^{11}$	921282.96
$2^{12}$	773445.95	1331398.38
$2^{13}$	629110.75	1198351.36	1334931.20
$2^{14}$	482549.41	1015442.16	1172515.95
$2^{15}$	327414.20	790934.41	970820.67
$2^{16}$	207518.83	568200.79	765018.06	1019024.18
$2^{17}$	119578.65	372601.88	549154.74	888694.10
$2^{18}$	65638.76	225165.28	363636.53	762964.26
$2^{19}$	34683.11	127139.76	221322.20	616003.37
$2^{20}$	17864.84	68460.39	127063.37	467957.75	667768.28
$2^{21}$	9100.34	35783.67	68607.02	325855.03	599878.21
$2^{22}$	4709.05	18519.38	36269.71	209302.84	533366.32
$2^{23}$	2372.26	9417.21	18696.17	123472.50	458117.32
$2^{24}$	1159.01	4709.63	9398.28	67672.98	357102.56

Batch read latency (us/batch), by batch size

N	B=1K	B=4K	B=8K	B=64K	B=1M
$2^{11}$	1096.72
$2^{12}$	1309.22	3016.52
$2^{13}$	1612.86	3359.04	6016.06
$2^{14}$	2107.44	3974.79	6866.85
$2^{15}$	3112.89	5119.80	8317.96
$2^{16}$	4919.82	7149.49	10587.80	63318.50
$2^{17}$	8548.79	10933.58	14797.73	72756.74
$2^{18}$	15585.89	18132.26	22407.66	84910.35
$2^{19}$	29509.56	32157.30	36892.67	105403.29
$2^{20}$	57304.28	59771.35	64350.53	139059.29	1554517.53
$2^{21}$	112508.16	114406.63	119284.25	200126.05	1732138.56
$2^{22}$	217436.17	221114.28	225741.28	312129.84	1949956.64
$2^{23}$	431636.06	434884.66	438042.22	529793.90	2272862.04
$2^{24}$	883498.55	869646.90	871527.41	967443.37	2920635.92

Load Balancer

8B keys, 8B values

Batch latency (us/batch), by batch size B

B	olabs_oram (S=15)	snoopy (S=15)
$2^{10}$	352.72	974.54
$2^{11}$	721.45	1882.36
$2^{12}$	1546.46	2795.21
$2^{13}$	3295.59	5023.23
$2^{14}$	6430.36	10177.94
$2^{15}$	11329.96	21697.00
$2^{16}$	22074.79	47132.77
$2^{17}$	47517.30	92517.58
$2^{18}$	96318.45	189689.73
$2^{19}$	195207.95	389596.66
$2^{20}$	394869.88	806894.73

Throughput (queries/s), by batch size B

B	olabs_oram (S=15)	snoopy (S=15)
$2^{10}$	2903191.25	1050749.06
$2^{11}$	2838715.84	1087996.15
$2^{12}$	2648634.55	1465364.47
$2^{13}$	2485747.54	1630822.62
$2^{14}$	2547913.40	1609756.72
$2^{15}$	2892154.49	1510254.58
$2^{16}$	2968815.93	1390454.97
$2^{17}$	2758405.89	1416725.31
$2^{18}$	2721638.42	1381961.98
$2^{19}$	2685792.22	1345719.97
$2^{20}$	2655497.54	1299520.20

Reproducing Benchmarks

Running Benchmarks

go.sh sets up and runs the benchmarks, saving full-run JSONL logs under logs/full_runs/:

sh go.sh

scripts/run.py wraps go.sh or parses its saved full-run logs into editable case lists or per-test timing/outcome CSV:

# List cases without running test bodies (setup/build still run):
BENCHMARK_MODE=plan uv run scripts/run.py > cases.txt
# Edit cases.txt: only run/yes entries execute; comment out cases to skip them.
# bulk update cases.txt: skip large N unless batch_size is 4096:
uv run scripts/edit_cases.py cases.txt \
  'N is not None and N >= 2**24 and batch_size != 4096' --in-place
BENCHMARK_SELECTOR=cases.txt uv run scripts/run.py > results.csv
# Run all encountered cases:
uv run scripts/run.py > results.csv
# Parse a saved run without executing anything:
uv run scripts/run.py logs/full_runs/FILE.jsonl > results.csv
# Generate a case list from a saved run:
uv run scripts/run.py --cases logs/full_runs/FILE.jsonl > cases.txt

Generating Figures and tables

Print all available result tables, or refresh only the curated tables already in README.md:

uv run scripts/generate_readme_bodies.py                     # All tables to stdout
uv run scripts/generate_readme_bodies.py --main              # README selection to stdout
uv run scripts/generate_readme_bodies.py --main --write-readme

--main preserves the current table selection, order, surrounding notes, and header footnotes while refreshing measured rows and columns. Delete a table to exclude it, or copy one from the full output to include it. Keep its section heading, details summary, and metric title unchanged; unmatched tables cause an error without writing. Omitting --main with --write-readme replaces the selection with the full output.

For figures:

uv run scripts/draw_figures.py

Both commands use the files list in scripts/plot_config.py to select benchmark results.

Repository Structure

Root Directory

• go.sh: Sets up and runs the benchmark pipeline.

benchmark/

Per-implementation code and setup.sh, build.sh, and run.sh scripts.

figures/

Contains PNG files visualizing benchmark results, such as:

• Initialization times
• Memory usage
• Latency and throughput metrics

logs/

Benchmark logs, including full-run JSONL events in full_runs/.

results/

Parsed benchmark results used by plots and tables.

scripts/

Utility scripts for managing and visualizing benchmarks:

• run.py: Wraps go.sh or parses full-run JSONL into case lists or per-test CSV.
• setup.sh: Fetches and builds benchmark implementations.
• reset.sh: Removes benchmark build directories.
• run_ramp_latency.sh: Runs latency workloads.
• parse.py: Extracts benchmark reports from logs into JSON results.
• draw_figures.py: Generates benchmark figures.
• generate_readme_bodies.py: Generates all Markdown tables, or refreshes the README selection with --main.
• plot_config.py: Selects result files for plots and tables.
• utils.py: Shared result-loading, plotting and table helpers.
• config_ubuntu.sh / config_arch.sh: Install system dependencies.

Contributing

Feel free to contribute to this repository by adding new benchmarks, improving scripts, or updating documentation. Take a look at CONTRIBUTING.md for details on how to contribute.