I'm picking up InceptionDB again after two months of focusing the free time I have left on getting flowtest.io off the ground (I like to think it will be the service that will revolutionize how APIs are tested).

Despite the time that has passed, I clearly remember that I was working on optimizing performance; I had the psychological goal of surpassing one million inserts per second (on a single node). The million wasn't hard to beat, but with AI I was determined to comfortably exceed the million. Actually, I was determined to push it a bit beyond the absurd and necessary, exploring various alternatives and attacking Inception's different abstraction layers.

I've repeated the latest tests again, and the result is so wild that it seems something is wrong. I select the first benchmark: INSERT, run it, and it finishes almost at the same time. It must be wrong! The log is clear: it inserted 1 million in less than 200ms, but I don't believe it. I think I must have left persistence disabled and it's only working in memory, or that there's no concurrency control and the data is overlapping, or that it counted the sent data but the server threw them to /dev/null.

sent: 1000000 took: 182.928406ms Throughput: 5466619.55 rows/sec Closing 'col-1779494946505865611'... http: Server closed

I'm suspicious. I can think of a thousand more reasons, but I put a break point right before deleting the database and run it again. Almost identical result despite debugging, but now I have this: /tmp/inceptiondb_bench_811457574/col-1779494946505865611 and it's not empty but only takes up 41MiB. Without thinking much, I move it to the data directory of my local Inception, start it, load the page, and there it is: 1M. And the data even seems correct.

Colección del benchmark con 1M de documentos
Benchmark collection with 1M documents

I still don't believe it. I create an index, run a query with ascending order, and find the first document, 0. I change the order and the last document, 999999, is also there. Everything seems in order. I stop paying attention to the screen to check that I'm still breathing.

It occurs to me to write a blog article, but first I repeat the test. This time with 10 million.

sent: 10000000 took: 1.078983549s Throughput: 9267981.90 rows/sec Closing 'col-1779498339775263815'... http: Server closed

Almost 10 million per second, which I estimate will be 410 MiB. Clearly still far from reaching the disk bottleneck that in real tests reaches 9.4 GiB/s; it doesn't even hit 5%.

The benchmark has several tests to measure performance in specific scenarios: INSERT, PATCH, REMOVE but there are others I didn't remember: INSERTPK, INSERTBTREE, RETRIEVEBTREE, RETRIEVEFS.

I'll run them all and record the performance of each one.

INSERT Throughput: 8785855.31 rows/sec Throughput Open: 8780487.73 rows/sec INSERTPK Throughput: 5890949.19 rows/sec Throughput Open: 2374067.17 rows/sec INSERTBTREE Throughput: 1766108.26 rows/sec Throughput Open: 3165092.19 rows/sec RETRIEVEBTREE Throughput: 5412335.80 docs/sec RETRIEVEFS Throughput insert: 9249222.80 rows/sec Throughput Full scan (no filter): 13145243.10 docs/sec Throughput Full scan with filter: 4265889.26 docs/sec PATCH Throughput: 2735628.98 rows/sec Throughput Open: 9973488.60 rows/sec REMOVE Throughput: 1622765.40 rows/sec Throughput Open: 14321483.85 rows/sec

In general, all operations exceed one million per second. Oh, and let me remind you that Throughput Open is the same operation but when starting the database. This is relevant because Inception relies solely on journaling to persist data, so to recover the state it has to read the entire journal before becoming operational again.