Sadiq Jaffer

Free Hardware Performance Counters in OxCaml Runtime Events

2026-04-04T19:37:00+01:00

At ICFP in Singapore last year I had lunch with Stephen Dolan and we got talking about hardware performance counters. Specifically, about how useful it would be to have them integrated into OxCaml's Runtime Events so you could ask questions like "which phases of my program have the lowest instructions-per-clock?" or "what is causing my TLB misses?". This post covers a prototype I've been working on since then.

OCaml's Runtime Events already gives you lightweight tracing of GC phases and user-defined spans in every OCaml binary. The prototype extends this to sample user-specified hardware counters at the start and end of each span, so you get performance data for free alongside the events you were already collecting. This works for built-in runtime events but also for custom events you can add around any logical section in your own code. Since spans can be continuously consumed, you can gather performance counter data on specific program or runtime phases across large numbers of long-running services. The overhead is indistinguishable from noise.

Why hardware performance counters?

Modern CPUs look like they execute instructions sequentially, but underneath they're parallel execution engines, reordering and speculatively executing instructions across deep pipelines. Multiple tiers of cache hide the fact that main memory is hundreds of cycles away and a single CPU might have dozens of cores that all need their caches kept coherent. To expose what's really going on, CPUs provide hardware performance counters. My 'Alder Lake' Intel i7-12700F has hundreds of them¹, covering everything from cache misses at each level to branch mispredictions to instructions retired.

On Linux the standard way to access these is through perf. You can run perf stat to get aggregate counter values for a whole program, or use perf record -e .. to do sampling based on counter overflow, which gives you a profile showing which lines of code are responsible for the event you're monitoring². These work well for standalone questions like finding the code causing the most L1 cache misses but they are unsuitable for continuous monitoring of long-running programs, and they can't easily correlate multiple counters with specific program phases, like asking which GC phases have the lowest instructions-per-clock.

Runtime Events already traces GC phases and user-defined spans in every binary. It also supports Custom Events, which let you emit spans and counters from your own code. If we extend it to capture hardware counters at span boundaries, we get continuous monitoring of both the runtime and user code, correlating hardware performance with logical program phases on real production traffic.

The implementation

The implementation has two sides: changes to the C runtime to set up and sample the counters, and changes to the OCaml API to expose the samples to consumers.

Runtime

We need to call perf_event_open for each counter we want to monitor on every thread and domain, and clean up when threads and domains exit. To handle this we add domain_start/stop, thread_start/stop hooks in the appropriate places in systhreads, covering both newly spawned threads and existing threads being registered.

When a Runtime Events span is started or stopped, we use rdpmc to sample the counters and append the values to the event data as it's written to the ring buffer. rdpmc takes around 20-40 cycles³, which is what makes the whole approach viable. That's very small compared to the useful work any span is doing.

The sampling is cheap but the configuration interface is less elegant. Counters to monitor are specified via the OCAML_RUNTIME_EVENTS_PERF_COUNTERS environment variable as a list of hex-encoded event codes and umasks. You need to look up the codes in the Intel Software Developer's Manual, from perfmon-events.intel.com or by running perf list and extracting the event code from perf stat -vv -e <counter name>.

For example, to count instructions, cpu-cycles, L2 cache misses (l2_request.miss) and L3 cache misses (longest_lat_cache.miss) on a Performance Core of a 12th Gen Intel Core i3-1220P: OCAML_RUNTIME_EVENTS_PERF_COUNTERS=r00c0,r003c,r3f24,r412e

OCaml API side

Consumers receive an additional local_ perf_sample array parameter on runtime_begin and runtime_end callbacks. This is an array of unboxed int64 pairs: the event code from the environment variable and the current counter value, in the same order as specified. The same parameter is available on add_user_event callbacks for custom events, and the samples are also accessible through the C API.

Taking it for a spin

The ahrefs benchmarks from Sandmark exercise code used heavily by their crawler and are known to be memory intensive, which is what we want for correlating GC behaviour with hardware performance. I had Claude port four of them (htmlStream, network, gzip, stre) to OxCaml, fixing up dependencies and using Anil's Oxmono to create a monorepo. You can find the benchmarks and the code to run them here: https://github.com/sadiqj/runtime-events-pmc-benchmarks.

GC phases by instructions-per-clock

Across all four benchmarks I monitored instructions and cpu-cycles for every runtime event span. The table below shows GC phases ordered by their total cycles, with the 5th, 50th and 95th percentile instructions-per-clock (IPC).

Phase	N	Cycles	%	IPC p5	IPC p50	IPC p95
major	10,728	4.91B	19.8%	0.237	1.748	3.994
major_slice	10,728	4.60B	18.6%	0.191	1.720	4.184
stw_leader	11,760	2.76B	11.2%	0.541	1.486	3.249
minor	6,093	2.59B	10.5%	0.650	1.955	3.917
minor_remembered_set	6,093	2.49B	10.1%	0.709	2.112	4.378
major_mark	2,622	2.38B	9.6%	0.612	2.639	5.029
minor_remembered_set_promote	6,093	2.27B	9.2%	0.614	2.150	4.748
major_sweep	6,216	2.19B	8.8%	0.096	1.762	4.317
major_gc_cycle_domains	1,689	288M	1.2%	0.917	1.706	3.359
minor_local_roots	6,093	85M	0.3%	0.411	1.145	3.823
major_mark_roots	1,680	65M	0.3%	0.859	1.233	1.987
minor_local_roots_promote	6,093	50M	0.2%	0.702	1.906	5.139
minor_finalized	6,093	28M	0.1%	0.763	2.051	2.444
major_gc_stw	1,689	16M	0.1%	0.837	1.438	2.415

The p5-to-p95 spread is wide across most phases. major_sweep ranges from 0.10 to 4.32 IPC, which likely reflects variability in dense versus sparse shared heap pools. A sparse heap pool could be difficult to branch predict, leading to pipeline stalls and low IPC. The next step to investigate would be to add a runtime/custom events counter inside the sweeping code that emitted the occupancy of pools. This would tell you whether the issue is sparse or dense pools, or possibly even the number of pools swept within a major_sweep phase.

minor_local_roots and major_mark_roots have the lowest median IPC at 1.15 and 1.23 respectively. Root scanning involves chasing pointers through the stack and globals which is unfriendly to caches and branch predictors, so low IPC is expected here.

major_mark has the highest median IPC at 2.64, with a p95 of 5.03. Marking does more computation per memory access than sweeping or root scanning, so the CPU has more useful work to keep its pipelines busy. There was also significant optimisation work done by Stephen Dolan on prefetching during marking.

Since consumers get counter values at the start and end of every span, you aren't restricted to looking at aggregates. You can dump out individual traces and inspect them too:

htmlStream — GC pause (211.8us, 46 phases, IPC 2.91)

IPC:0

4.5+

0ns

25us

51us

76us

102us

127us

152us

178us

203us

229us

254us

stw_leader — 128.5us — IPC 1.90

minor — 112.0us — IPC 2.02

m_remembered_set — 110.4us — IPC 2.03

m_rs_promo — 109.1us — IPC 2.05

M_mark_roots

major — 58.6us — IPC 5.09

M_slice — 51.7us — IPC 5.54

M_mark — 50.1us — IPC 5.67

stw_leader

major — 23.8us — IPC 3.77

M_slice — 23.5us — IPC 3.79

M_sweep — 23.3us — IPC 3.79

Beyond the GC — profiling your own code

Performance counters are also emitted for Custom Events. Adding a custom event around a section of code is straightforward:

Runtime_events.User.write span_name Begin;
(* your code here *)
Runtime_events.User.write span_name End;

We added custom events around each benchmark phase in the ahrefs benchmarks. This time I monitored L2 and last-level cache (LLC) misses alongside cycles. The table below shows LLC misses and L2 misses per 1,000 instructions:

Phase	N	Cycles	%	LLC/ki	L2/ki
bench:large_blocks	3	54.89B	32.5%	0.8	1.7
bench:generational_violation	3	29.77B	17.6%	0.6	2.6
bench:morphing_heap	3	14.81B	8.8%	0.6	1.5
bench:complex_network_ops	3	10.62B	6.3%	5.6	12.7
bench:variable_rate	3	10.49B	6.2%	1.8	5.0
bench:range_operations	3	7.99B	4.7%	0.2	1.5
bench:attribute_lists	3	6.51B	3.9%	0.5	2.3
bench:complex_references	3	5.83B	3.4%	0.8	3.1
bench:fragmentation	3	4.50B	2.7%	0.6	1.6
bench:cidr_calculations	3	3.25B	1.9%	0.4	3.7
bench:small_buffer_storm	3	3.21B	1.9%	0.4	1.3
bench:mixed_size_patterns	3	2.99B	1.8%	0.5	1.6
bench:compression_pipelines	3	2.63B	1.6%	0.4	1.6
bench:ip_sorting	3	1.81B	1.1%	0.4	3.2
bench:large_block_compression	3	1.61B	1.0%	0.3	1.2
bench:split_storm	3	1.30B	0.8%	0.6	3.4
bench:streaming_operations	3	1.16B	0.7%	0.1	1.9
bench:buffer_recycling	3	1.06B	0.6%	0.5	3.0
bench:broadcast_calculations	3	925M	0.5%	0.6	4.5
bench:ipv4_parsing	3	855M	0.5%	0.6	3.8
bench:concurrent_style	3	553M	0.3%	0.3	0.9
bench:small_strings	3	432M	0.3%	0.3	1.3
bench:pattern_operations	3	384M	0.2%	0.7	4.9
bench:mixed_format_parsing	3	350M	0.2%	0.5	3.2
bench:nat_tables	3	250M	0.1%	1.1	10.7
bench:headers_metadata	3	235M	0.1%	0.8	3.1
bench:enum_string_ops	3	201M	0.1%	0.3	1.4
bench:mixed_size_allocations	3	201M	0.1%	0.5	2.4
bench:string_building	3	127M	0.1%	0.8	4.6
bench:transformation_chains	3	94M	0.1%	0.5	4.0
bench:concatenation_chains	3	32M	0.0%	0.3	4.0
bench:substring_slicing	3	6M	0.0%	0.4	5.2

Most phases have very low LLC miss rates, well under 1.0 per 1,000 instructions. bench:complex_network_ops is the exception at 5.6 LLC misses per 1,000 instructions, roughly 10x the rate of most other phases. Its L2 miss rate of 12.7 is also the highest in the table, so if you were chasing memory access patterns this is where you'd start.

bench:nat_tables stands out too. Its LLC miss rate of 1.1 is modest but its L2 miss rate of 10.7 is among the highest, suggesting a working set that fits in LLC but not in L2.

With perf alone you would need to run separate profiling passes for each counter and then try to correlate them with specific sections of your code. Here you get it all in one continuous stream.

Like the runtime events, we can look at individual traces. Here are some operations from the gzip benchmark, showing user events alongside runtime ones. You can mouseover for more detail:

gzip / streaming_operations — GC interrupting application sub-phases (284.7us, 54 spans)

IPC:0

4.5+ thick border = application span

0ns

28.5us

56.9us

85.4us

113.9us

142.3us

170.8us

199.3us

227.7us

256.2us

284.7us

streaming_operations — 91.6ms — IPC 3.68

stream_chunked_w

stream_chunked_r

stream_chunked_write — 43.2us — IPC 2.99

stw_leader — 35.1us — IPC 0.86

minor

M_mark_roots

m_finalized

major

major — 36.0us — IPC 1.30

M_slice — 35.7us — IPC 1.29

M_sweep — 35.4us — IPC 1.30

stream_chunked_r

stream_chunked_write 3.55

stream_chunked_r

stream_chunked_w

How expensive is it?

Each benchmark was run three ways: with runtime events off entirely, with runtime events on but no performance counters, and with runtime events on and four performance counters. Each configuration was run 20 times, pinned to a single P-core on my i3-1220p server, measuring wall-clock time.

Benchmark	Baseline	Events Overhead	PMC Overhead
htmlStream	29.26 +/- 0.16s	-0.0% [-0.5, 0.3]	+0.0% [-0.5, 0.5]
stre	5.32 +/- 0.04s	+0.1% [-0.4, 1.1]	+0.5% [-0.0, 1.5]
network	6.03 +/- 0.16s	+0.1% [-1.1, 0.5]	+0.0% [-0.6, 1.0]
gzip	2.79 +/- 0.02s	+0.3% [-0.2, 0.9]	+0.4% [-1.6, 1.6]

The 5th and 95th percentiles are in square brackets. In every case the range crosses zero, which means the overhead is not distinguishable from noise. The numbers confirm what the cycle counts predict, that rdpmc adds around 30 cycles per counter per span. Even with four counters that's roughly 120 cycles at each span boundary, which is tiny compared to the work any GC phase is doing. The longest benchmark here runs for nearly 30 seconds.

This makes it practical to leave performance counters enabled on long-running services and collect hardware-level data continuously.

Trying it yourself

Note that this is currently x86-only and Linux-only since it relies on rdpmc and perf_event_open.

To install the prototype, create an opam switch from the branch:

opam switch create oxcaml-dev --empty
opam pin add -ny git+https://github.com/sadiqj/oxcaml#runtime_events_pmc
opam switch set-invariant -y --packages oxcaml-dev
eval $(opam env)

Build your program as normal. To specify which counters to monitor, set OCAML_RUNTIME_EVENTS_PERF_COUNTERS with the hex event codes as described above.

You'll also need a consumer to read the events. There's an example consumer in the benchmarks repo that connects to a running program given its executable directory and PID, and writes a CSV of all events with their durations and counter values.

If you try it out, let me know what you find.

Next steps

This is a prototype and isn't merged upstream to OxCaml yet. There are a few things I want to address before proposing it:

The counter configuration needs to be less clunky. Requiring hex event codes works but ideally you'd be able to specify counters by name, similar to how perf lets you write cache-misses instead of r412e.

The API could also be cleaner. Right now the performance samples are passed as an additional parameter on every callback, which makes the signature noisier even when you don't care about counters.

I'd like to explore the arm64 equivalents. arm64 has comparable facilities through the PMU.

Finally it would be worth extending Olly to support collecting the counters and pushing them through to traces, which would make this accessible without writing a custom consumer.

Longer term, the fact that the overhead is negligible means you could leave counters enabled across an entire fleet of services. Aggregating hardware counter data across hundreds of instances would let you ask questions that are impossible on a single machine, like which program phases have the worst cache behaviour on a specific microarchitecture, or whether certain workloads cause specific GC phases to degrade.

In reality it has nearly twice as many because there's one set for the P-cores and one set for the E-cores - you have to be reasonably careful to pin to a specific set of cores. ↩
This isn't entirely true because as we mentioned earlier, modern CPUs are parallel execution engines, so what does it even mean when an event happens but there's 100s of instructions in-flight? Sometimes we've already executed a number of instructions before the interrupt that captures the stack. A subset of the Intel counters are able to more precisely determine their causing instruction. I recommend these articles. ↩
Agner Fog has cycle counts in his excellent data tables ↩

Earth Observation on a Budget: Finding Solar Farms with a 42k-Parameter Model

2026-01-14T17:38:00+00:00

Solar farms are expanding rapidly across the UK, but keeping track of where they are, and when they appeared, isn't straightforward. In this post, we'll map them using satellite imagery and a surprisingly small neural network with only ~42k parameters.

We're going to do this using the Tessera foundation model, a pre-trained AI model that already understands satellite imagery in a similar way to how Large Language Models 'understand' text. Using Tessera is a little different from a Large Language Model in that you don't interact with the model directly but rather use pre-generated embeddings for the areas you want to analyse. These embeddings are available at 10m resolution and form a rich summary of that point on Earth's light and radar reflectance over a calendar year, its temporal-spectral characteristics.

We actually used Tessera when we went hunting brambles a few months back. In that instance, we created a pixel-wise model to predict whether a specific 10-meter square contained brambles. This is very easy to do with the efficient sampling support in Geotessera and works fine when the entity is defined purely by its surface temporal-spectral characteristics. But what if the entity you are looking for is defined by its shape and surroundings? For example, a single pixel might be asphalt but whether it is part of a car park or a road must take into consideration its surroundings. In this post we're going to classify solar farms, which exhibit this property. Despite having similar spectral characteristics what separates a solar farm pixel from a rooftop solar install is the size and shape of the solar panels around them.

We could simply classify whether a 1km tile contains a solar farm. However, it's usually more useful to map the exact shape of the farm pixel-by-pixel. This technique is called semantic segmentation, a specific type of dense prediction.

So step one, we need to find some solar farms. For this we use two data sources: OpenStreetMap and the UK government's Renewable Energy Planning Database. We join the OSM solar farm polygons with the REPD to find farms that have an operational date in REPD or a start date in OSM before 2024, this helps filter out farms built in 2024 or 2025. We want to find solar farms that existed throughout 2024.

Now we have a set of solar farm polygons, we need to divide this into train, validation and test sets. For this we use a 25km checkerboard split with roughly 78%, 11% and 11% train, validation and test respectively. We split this way to avoid leaking data from partial and neighbouring polygons. This results in 760 train polygons, 112 validation and 118 test.

Checkerboard spatial split pattern

We then produce a set of tiles around each polygon, each 160x160 (so 1.6km x 1.6km). As the Tessera embedding is 128 dimensions, this effectively gives us 128 channel images. Here are a couple with just the first 3 channels of the Tessera embeddings.

Tile with first three Tessera dimensions (hover to highlight solar farm)

For training targets, we have a binary true/false for each pixel in the image depending on whether it is contained within a solar farm polygon or not.

Now we have these tiles we can train a model using them. For this we use a Convolutional Neural Network (CNN), specifically a UNet. This network design has an encoder-bottleneck-decoder structure with skip connections between the stages and crucially lets us do dense prediction, that is, gives us a prediction at every point. We use a nano-sized UNet configuration, which is only ~42k parameters. It may seem surprising to use such a small model but we've found the richness of Tessera's embeddings means we often don't need very large models. Smaller models mean fewer parameters to fit and in general, less training data. They're also much faster to train, this nano-sized model can be trained in about 15 minutes on CPU alone.

To train the model we use a combination of the Dice loss and the Binary Cross Entropy (BCE) loss. The Dice loss encourages the model to overlap the predicted solar farm regions with our ground-truth regions whilst BCE penalises the model pixel-by-pixel for being wrong, this survey paper has a list of all the possible losses one could use and their properties. We also apply label smoothing (a technique that makes the model less confident in its predictions), which helps it tolerate labelling errors, the kind we probably have through our assumptions about the negatives surrounding solar farms as well as any missing solar farms in OSM that might be in our training tiles.

A final trick is that we 'stretch' our training data through augmentations: random cropping, flipping, and rotating the tiles. This forces the model to learn the essential features of solar farms rather than memorising specific orientations or positions. I was concerned that the orientation of solar panels was actually a strong signal and the rotational and flip augmentations would be harmful but they seem to give a small benefit instead.

Training this takes less than 4 minutes on a Ryzen 9950X + NVIDIA RTX 5090 machine. I'm fairly sure this could go faster with a bit more optimisation. To evaluate the model we focus on the Intersection-over-Union (IoU) which measures the overlap between the model's prediction and the ground truth and divides this by their combined union. The trained model reaches 0.76 IoU on the evaluation set, with a precision of 0.80 and recall of 0.93, which isn't bad at all for a model where we've had to make some strong assumptions around our training data.

Now we have a small trained model, we can use it to do inference. I've run the model over 2017-2024 in the UK. Here's some animations showing growth of solar farms (many of these existed before 2017) in Cornwall and Devon:

Solar farms 2017-2024 in Cornwall

Solar farms 2017-2024 in Devon

(Both of the above use cloudfree composites from https://s2maps.eu by EOX IT Services GmbH which contains modified Copernicus Sentinel data 2017-2024)

Here's a picture of a chunk of England and Wales, the black areas are solar farms the model identified:

Solar farms detected by the model across England and Wales

Note: You can also download the model-generated geojson for the whole of the UK, though this has not been validated and almost certainly contains some level of false positives.

Lastly, inferring tiles that are not in the train/validation/test set has revealed a few detected solar farms that don't seem to be in OSM. Determining that is tricky from Sentinel 2 alone, once we have access to a higher resolution source of imagery we should submit the missing ones.

This demonstrates that effective Earth observation doesn't necessarily require massive compute resources. By leveraging the rich, compressed information in Tessera's embeddings, we were able to train a model with only ~42k parameters to identify solar farms across the UK.

There's a general recipe here for using Tessera to detect almost anything that's visible from space:

Get your Ground Truth: Start with a dataset of polygons for the thing you want to find. We used OpenStreetMap and the REPD, but you could use any GIS database or even hand-label a small region.
Split Spatially: Don't just split your data randomly into train/validation/test sets. Use a spatial split (like our 25km checkerboard) to ensure your model learns to generalize rather than just memorizing the landscape. This is especially important when you are doing classification in very dense areas and where entities can end up very close to each other.
Start with the smallest model you can, it may surprise you: As we saw in our solar farm example, even a ~42k parameter model produced good results. We found in our Tessera paper even linear models performed reasonably. This makes inference very cheap, when inferring over large landscapes you may find your bottleneck is actually bandwidth.

Give it a go and let me know what you find.

Three Steps for OCaml to Crest the AI Humps

2025-10-24T17:38:00+01:00

I gave the talk "Three Steps for OCaml to Crest the AI Humps" at the 2025 OCaml Workshop at ICFP/SPLASH. In it I discussed the challenges and opportunities for OCaml in the age of AI coding assistants.

You can find a video of the talk here. There's also an extended abstract. This blog post has a copy of the slides along with some commentary taken from the speaker notes and other links.

OCaml in training data: ~0.003% of The Stack v2

OCaml represents approximately 0.003% of The Stack v2, a major code corpus used for training large language models. Compare this to Python, which makes up about 8% of the same dataset. There are consequences for having so little data available..

5x performance difference between Python and OCaml

The disparity translates in a difference in problem solving ability between languages. Cassano et al found a 5x difference in Pass@1 HumanEval performance using Starcoder 15B between OCaml and Python. This performance gap has real consequences for developer productivity.

An existential threat to smaller language communities

The gap between coding agent performance on mainstream versus niche languages could prove fatal to smaller language communities. New developers increasingly judge a programming language not just on its traditional tooling (compilers, debuggers, libraries) but also on how well AI coding agents support it. If agents struggle with OCaml, fewer new developers will choose to learn it, creating a vicious cycle.

Three steps forward

Rather than accept this fate, there are concrete steps the OCaml community can take to improve AI agent support. The talk outlines three actionable strategies that work together to address the training data gap and knowledge accessibility problems.

Step 1: Where are the deficiencies?

The first step is to systematically determine where AI agents are deficient for your specific, real-world use cases. You can't improve what you don't measure, and generic benchmarks might not capture the specific ways your community uses the language.

Define success for your use case

To measure agent performance, you first need to define what "well" means for your particular use case. Everyone has different use cases e.g sync vs async, hosted vs local, fine-tunable.

Our specific needs

For our evaluation at Cambridge, we had specific requirements: the models needed to run locally or on university servers (no reliance on external APIs), support teaching environments (helping students learn, not just solving problems), be reproducible and have provenance, and be fine-tunable for our specific needs.

Our benchmark approach

We chose to evaluate models using the first-year Computer Science Jupyter exercises ("ticks") from Cambridge's Foundations of Computer Science course. These exercises measure introductory CS problem-solving skills in OCaml and are directly relevant to our use case of supporting student learning. The exercises test concepts like recursion, data structures, and streams.

Example: Finding subscripts of even elements

This slide shows an actual exercise from the benchmark. Students need to write a function getSubsOfEvens that returns the subscript positions of even integers in an array. The notebook provides test cases and automatically checks solutions - the same setup we used for evaluating AI models.

More exercises: Binary trees and sets

The exercises cover a range of computer science fundamentals. Here we see problems involving binary search tree operations (intersection) and set operations (remove). These test both algorithmic thinking and familiarity with OCaml's syntax and type system.

Results: Model parameter count vs. success rate

We benchmarked 19 different open-weight models, from 7B to 70B parameters. Within a family, larger models generally perform better, but there are interesting outliers. The Qwen3 family of models consistently perform well especially when using "thinking" mode. Claude-3.7-Sonnet (96.4%) sits at the top as our reference point for proprietary models.

Key finding: Qwen3 performance with reasoning

Qwen3-32b with thinking mode achieved 94.2% success rate, nearly matching Claude-3.7 Sonnet's 96.4%. Without thinking mode, the same model scores only 62.4%. This demonstrates that reasoning significantly improves performance on OCaml tasks, and that, on this task, self-hostable models can achieve near-frontier performance when given space to reason. The cost is increased latency and higher token usage.

Common mistakes AI models make

Smaller models frequently make syntax errors. Type system confusion is very common - using integer operators (like +) on floats is a recurring issue. Models hallucinate functions that don't exist in the standard library, such as List.sub or List.combinations, and often assume the Core library or Format module are available when they're not. Recursion errors are also frequent: forgetting the rec keyword, using incorrect base cases, or failing to make recursive progress.

Step 2: Contribute to benchmarks

Once you've identified deficiencies for your use cases, the second step is to contribute those tasks to public benchmarks. This is how you signal to AI labs what matters to your community.

Add OCaml use cases to public benchmarks

AI organizations report performance on public benchmarks like SWE-bench and terminal-bench when releasing new models. They actively train their models to improve on these benchmarks. By adding OCaml-specific tasks that you care about to these benchmarks, you create an incentive for labs to improve their OCaml capabilities. If OCaml tasks are on the benchmark, models will get better at OCaml.

Example: OCaml GC bug fix task

As a concrete example, I spend a significant amount of time debugging the garbage collector in OCaml and I found frontier models were still very poor at aiding me do that. So I contributed a task to terminal-bench that requires an agent to debug a segmentation fault in the OCaml garbage collector. In hindsight I worry this was a short-sighted career move (since I'm teaching agents to replace my debugging work).

What the GC debugging task requires

The task is genuinely difficult and tests multiple capabilities: (1) build the OCaml compiler and runtime, (2) reproduce a GC failure, (3) debug with gdb, (4) patch C code, and (5) verify the fix. Completing this requires reading HACKING.adoc to understand the build system, navigating OCaml's runtime C code, and using debugging tools effectively - exactly the kind of complex, multi-step reasoning we want agents to excel at.

The agents are finding it challenging

The task is proving quite difficult for agents. As evidenced by these GitHub PRs, the timeout for the fix-ocaml-gc task has been increased multiple times because agents are struggling to complete it within the time limit. This is actually good - it means the task is pushing the boundaries of current agent capabilities and driving improvement.

Continuous effort required

Contributing to benchmarks isn't a one-time activity. It needs to be done constantly because benchmarks eventually get "saturated" (models overfit to them) or are leaked into new training sets. The community needs to continuously create new, challenging tasks that represent real OCaml development work to maintain pressure on AI labs to improve their OCaml support.

Step 3: Tools and documentation

The third step addresses the immediate problem: even if we can't add more OCaml to training data, we can bridge the knowledge gap by providing good agent-friendly tools and documentation. This can compensate for the lack of training exposure.

Two complementary approaches

There are two main approaches: agent-friendly documentation and tools built on the Model Context Protocol (MCP). Documentation needs to be clean and parseable, while tools give agents the ability to search, discover, and navigate the OCaml ecosystem programmatically.

What humans see: clean documentation

When we visit ocaml.org documentation, we see nicely formatted pages with the information we need. The HTML rendering makes it easy for humans to navigate and understand the API documentation.

What agents see: HTML soup

The problem is that when an agent accesses the same documentation, it sees a mix of content and messy HTML markup. This wastes valuable space in the model's limited context window and makes it harder to extract the relevant information. The signal-to-noise ratio is terrible for agents trying to understand OCaml APIs.

What we should provide: clean markdown

The solution is to provide clean, markdown-only versions of documentation. This gives agents the information they need without the HTML clutter. Recent improvements to odoc have merged the capability to generate markdown output, and emerging standards like LLMs.txt files provide additional ways to make documentation agent-accessible.

odoc-llm: MCP tools for OCaml

Beyond documentation, we can provide tools that agents can actively use. Along with Jon Ludlam I've built a prototype called odoc-llm that uses the Model Context Protocol to give agents a set of tools for interacting with the OCaml ecosystem. These tools allow agents to search for packages by functionality, get concise summaries of modules, and search through local project documentation using Sherlocode. This is critical for helping agents discover and use packages they were never trained on. The search uses a hybrid dense and sparse approach. Finding functionality in the OCaml ecosystem is genuinely hard, even for humans.

Recap: Three actionable steps

To summarize: the OCaml community can fight back against the existential threat posed by poor AI agent support through three concrete steps. First, determine where agents are deficient for your specific use cases through systematic evaluation. Second, contribute those use cases to public benchmarks to incentivize AI labs to improve. Third, bridge the knowledge gap with agent-friendly documentation and tools that compensate for the lack of training data. With these steps we can try to close the performance gap.

Can a model trained on satellite data really find brambles on the ground?

2025-09-24T21:04:00+01:00

Over the summer Gabriel Mahler has been conducting research on hedgehog habitat mapping using Agent Based Models (ABMs) and remote sensing. Hedgehogs seem to like brambles and so as part of his work he has produced a bramble map. He did this by combining the TESSERA earth representation embeddings (using the geotessera library) with data from iNaturalist. The current model is an ensemble of logistic regression and a knn classifier.

Can we really see brambles from space? What better way to test the model than a quick field trip around Cambridge. Gabriel, Anil, Shane and I did just that today.

Gabriel's model overlaid on a map - which was quite hard to read in outdoor sunlight

We started at Milton Community Centre, as the model was relatively confident there were brambles near the car park and along the path to Milton Park. It took us about 20 seconds to find the first one in an area indicated by the model.

Our first bramble!

So it turns out that there's a lot of bramble between the community center and entrance to Milton Country Park. We stopped six or seven times before reaching the park entrance. While the model predicted we'd find brambles all over the park, we went for the few areas of very high confidence near the entrance. In every place we checked, we found pretty significant amounts of bramble.

Some bramble on both sides of the path near the park entrance

Trying to navigate using the model

I was fairly sure it was in that direction (and thankfully it was)

We collected photos of all the places we stopped, as well as recording our GPS location. One thought while out exploring is that the model did a great job predicting where we would find very large quantities of bramble without any cover. It didn't have high confidence in other areas where we found smaller brambles under partial cover. Since TESSERA is learned representation from remote sensing data (Sentinel 1 and 2), it would make sense that bramble partially obscured from above might be harder to spot. This is something we can potentially tease apart when we have more validation data.

And more bramble

Stopping to check the model

Stopping to take a photo of a very photogenic bee

Finally, we were satisfied the model was doing a good job in the park area and decided to pick a hotspot the model was predicting in part of a residential street. We drove over to find an empty plot that did indeed have a lot of bramble!

Empty area that had a lot of bramble

Another hotspot was on Fen Road and we stopped by to find this absolute unit:

Absolute unit of bramble

Finally, we headed back in to Cambridge to see what one of the big hotspots in North Cambridge was like. To our amusement we ended up at the local nature reserve Bramblefields, which, true to its name, has a lot of bramble.

Bramblefields

I was pleasantly surprised by how good Gabriel's model was for its simplicity. Great work!

We had hoped to actually re-run the model based on the data we were gathering but that proved tricky on a laptop, in a park. Given the richness of the TESSERA embeddings and the simplicity of the classifiers being used, a mobile phone-based human-in-the-loop active learning setup could be practical..

Adding an OCaml GC debugging task to terminal-bench

2025-08-20T10:01:00+01:00

I stumbled onto terminal-bench a few weeks ago while researching datasets to evaluate agents. It contains around 120 tasks that need to be completed using a terminal, they range from easy things like fixing permissions to more tricky things like finding lost changes in git and installing windows xp (!).

To that collection, we can now add a freshly-merged task focused on finding and fixing a bug in the OCaml GC. This task is designed to be difficult; it involves a subtle change that isn't obvious at first pass even for experts and mirrors a similar real issue I encountered earlier in the year. Solving it requires interactive debugging.

As more model providers report performance on terminal-bench, it's a great opportunity to add tasks for common development workflows that aren't currently covered. This could include more native debugging tasks, as well as some general sysadmin tasks - though the latter might be tricky because the containers tasks run in are unprivileged. I'm sure Mark Elvers has some good ideas..

Last three months in OCaml (July 2025)

2025-07-15T10:01:00+01:00

Thanks to sponsorship from Tarides I've been able to spend some time over the last few months hacking on various OCaml-related projects. Some of these I've already published as blog posts, while others are still works in progress but this post gives brief summaries and ties some of them together for a bigger picture.

AI Coding Agents

Jon Ludlam, Anil Madhavapeddy and I have spent a fair amount of time thinking broadly about AI coding agents. In addition to this post it's worth checking this short draft paper we've written that summaries our work so far and our thoughts for what could be needed going forward for OCaml to work well with AI tooling.

Benchmarking

Anil and Jon lecture the first year Foundation of Computer Science here at Cambridge, so we tested how well self-hostable models performed on first year coding exercises. We found the Qwen3 models were very effective, rivaling much bigger and older models. We plan to do some more benchmarking, especially as we've heard good things from community members about newer models like Mistral's devstral which has just been updated. Before we can do that though, we want to build a more extensive set of benchmarks. A step in that direction is opam-archive-dataset which is a continuously-updated parquet dataset of all source code in opam. We've also been looking at SWE-Synth and how we would apply a similar method for synthesising across community OCaml projects.

Agentic tooling

There are coding models that we can't modify ourselves¹ and which users access remotely. Nearly all agents using these coding models support tool use through the Model Context Protocol. We've started working on a tool odoc-llm which enables natural language search of functionality over all packages and libraries in opam - in a way that can be hosted centraly as a remote MCP server. It's still a work in progress and Jon is planning to write something more extensive once we've fixed the last few issues. We've mentioned a few other potential tools in the draft piece and if anyone in the community is working on any of them or would like to collaborate then please do get in touch!

I've also been exploring using simple collaboration between coding models and created a very simple MCP server that lets you use gemini-cli as tool in another agent. Mostly I use this to have Claude Code call Gemini to check it's work - this can be helpful for tasks that involve analysing larger code bases where Gemini's larger context window is useful.

The OCaml runtime

In addition to the forward-looking work around AI models I've also been doing a bit of runtime maintenance. The change to the shared heap's free list representation has gone through two stages of review and should land fairly soon. We're just waiting on some final performance testing before merging.

I had a fun debugging session with Jan Midtgaard trying to figure out what could have caused ocaml issue 13739. It turned out that terminating domains could orphan shared pools in parallel with a running stop-the-world section, which was unexpected and bad as it could lead to a segfault or memory corruption if you were very unfortunate. Gabriel Scherer fixed this in ocaml pr 14025.

Lastly with the GC there is the long-running compactor unification between the trunk ocaml compactor and OxCaml compactor². The OxCaml compactor is a bit more expensive than trunk but plays much better with virtual memory. This helps reduce memory usage and improve performance in large, long-running applications. I suspect proposing the OxCaml compactor for trunk will be something worth doing next quarter.

Finally, there have a number of small fixes to runtime events. It's nice to see these and new feature requests appearing.

Projects

I had hoped to kick off a couple of undergraduate or masters projects along with KC Sivaramakrishnan, either at Cambridge or at IIT Madras. These are intended to be ways of mentoring students who think they might like to become contributors to the OCaml runtime. Unfortunately, February and March are a bit late for picking projects on academic courses.

The first is implementing 'Early Release' for the OCaml minor collector. At present, all domains pause for a minor collection and wait until all domains have finished promoting their minor heaps before resuming. Consider each domain, it's minor heap consists of values that are reachable from other domains but also values that only it can reach. If all domains have promoted the global roots, their local roots, and their remembered set, then any domain that has finished promoting everything in their minor heap should be free to leave the minor collection and resume the mutator. This should mean that domains that do few minor heap allocations spend very little time in minor collections. We had an early prototype of this implemented a few years ago on the ocaml-multicore repo but we weren't convinced it was correct and ran out of time to fix it before upstreaming multicore. I think now is a good time to try again.

The second project is 'Domain runtime work-sharing'. At present the number of domains needs to be less than the number of available physical cores. Going over that number can significantly reduce performance as domains wait to enter or complete a stop-the-world section in the runtime. Could we address this by restricting the number of domains running in a stop-the-world section to the number of physical cores and have that set of domains take on the work of all stopped domains? This is probably most advantageous for minor collections but could be done for major stop-the-world sections too.

A third wild card project is exploring how we could use information from the GC and Linux's sched_ext to better schedule highly parallel OCaml programs. This is a lot more speculative but might make for an interesting Masters project.

If you are interested in any of these projects for the next academic year, please do get in touch with me or KC.

Mainly models from proprietary providers like OpenAI, Anthropic and Google or where the models are huge and uneconomical to run in anything less than large multi-user deployments, like Deepseek V3/R1 or Kimi 2. ↩
OxCaml actually has two compactors, the trunk compactor and a new one. I only mean the new one here. ↩

Letting Claude Code check with Gemini

2025-06-27T10:01:00+01:00

gemini-cli was released a few days ago with a very ample free tier that gives access to Gemini 2.5 Pro, a model with a million token context window. I'm already a user of Claude Code so I wondered if I could have the two talk to each other. Enter check-with-gemini which is a simple mcp server and lets Claude Code call Gemini to help analyse codebases or check plans.

The MCP server has a single tool check_with_gemini which takes a prompt and content. Here's some examples of it in use.

Examples

I asked Claude Code to explain a codebase to me without Gemini:

> Analyse this codebase in depth. Produce a short two paragraph summary for a potential new contributor.

● I'll analyze this codebase to understand its structure and purpose, then provide a summary for potential contributors.

● Task(Analyze codebase structure)
  ⎿  Done (10 tool uses · 44.6k tokens · 1m 28.6s)                                                                                                                                                                  

● This codebase is an AI Pipeline Visualization project that creates an interactive 3D visualization of academic papers and their hierarchical clustering relationships. Built with C++20 and OpenGL 4.1, it renders
   thousands of papers as animated points in 3D space, grouped into transparent convex hull clusters representing semantic topics. The system loads paper data from CSV files containing titles, coordinates, and
  cluster assignments at multiple hierarchical depths (4-64 clusters), then provides real-time animation showing paper exploration progress with interactive camera controls and dynamic statistics.

  For contributors, the project follows a clean architecture with clear separation of concerns: PaperLoader handles data parsing, ClusterRenderer manages 3D visualization, and the OpenGL framework provides
  efficient instanced rendering. The build system uses CMake with dependencies on GLFW3, GLM, FreeType2, and Assimp. Key areas for contribution include improving Windows platform support (currently incomplete),
  optimizing rendering performance for larger datasets, extending the UI with additional visualization modes, or enhancing the clustering algorithms. The codebase is well-structured with logical file organization
   under src/ for core components and data/ for assets, making it approachable for developers familiar with modern C++ and OpenGL programming.

Now with check_with_gemini:

> Analyse this codebase in depth. Check with Gemini that your understanding is correct. Produce a short two paragraph summary for a potential new contributor.

● I'll analyze this codebase in depth and verify my understanding with Gemini before providing a summary for potential contributors.

● Update Todos
  ⎿  ☐ Explore project structure and identify main components
     ☐ Analyze build system (CMakeLists.txt)
     ☐ Examine main.cpp and core functionality
     ☐ Check with Gemini for understanding verification
     ☐ Write contributor summary
     ☐ Review recent commits for project direction

● Update Todos
  ⎿  ☐ Explore project structure and identify main components
     ☐ Analyze build system (CMakeLists.txt)
     ☐ Examine main.cpp and core functionality
     ☐ Check with Gemini for understanding verification
     ☐ Write contributor summary
     ☐ Review recent commits for project direction

● List(.)
  ⎿  Listed 1119 paths (ctrl+r to expand)

● Search(pattern: "**/*")
  ⎿  Found 100 files (ctrl+r to expand)

● Read(README.md)
  ⎿  Read 65 lines (ctrl+r to expand)

● Update Todos
  ⎿  ☒ Explore project structure and identify main components
     ☐ Analyze build system (CMakeLists.txt)
     ☐ Examine main.cpp and core functionality
     ☐ Check with Gemini for understanding verification
     ☐ Write contributor summary
     ☐ Review recent commits for project direction

● Read(CMakeLists.txt)
  ⎿  Read 92 lines (ctrl+r to expand)

● Read(main.cpp)
  ⎿  Read 557 lines (ctrl+r to expand)

● Read(csv_loader.cpp)
  ⎿  Read 174 lines (ctrl+r to expand)

● Update Todos
  ⎿  ☒ Explore project structure and identify main components
     ☒ Analyze build system (CMakeLists.txt)
     ☒ Examine main.cpp and core functionality
     ☐ Review recent commits for project direction
     ☐ Check with Gemini for understanding verification
     ☐ Write contributor summary

● Bash(git log --oneline -10)
  ⎿  903a1c0 Start animation paused
     ab728d8 Reduce vignette
     59cb10e Update README.md
     … +7 lines (ctrl+r to expand)

● Read(src/paper_loader.h)
  ⎿  Read 122 lines (ctrl+r to expand)

● Read(src/clusters.h)
  ⎿  Read 112 lines (ctrl+r to expand)

● Update Todos
  ⎿  ☒ Explore project structure and identify main components
     ☒ Analyze build system (CMakeLists.txt)
     ☒ Examine main.cpp and core functionality
     ☒ Review recent commits for project direction
     ☐ Check with Gemini for understanding verification
     ☐ Write contributor summary

● check_with_gemini:check_with_gemini (MCP)(prompt: "Review this analysis of a C++ OpenGL visualization project for scientific papers. Is my understanding correct?\n\nThis is an AI Pipeline Visualization project
                                           that:\n1. Loads scientific paper data from CSV files containing paper titles, inclusion status, 2D/3D positions, and hierarchical cluster assignments (clusters at depths
                                            2-6)\n2. Visualizes papers as instanced 3D cubes using OpenGL with diffuse/ambient lighting\n3. Groups papers into hierarchical clusters represented by transparent
                                           convex hull models\n4. Provides interactive exploration with animation that progresses through papers chronologically\n5. Features include bar charts showing cluster
                                           distributions, viewing mode toggles, and animation speed controls\n6. Uses modern C++ (C++20), CMake build system, and libraries: GLFW3, GLAD, GLM, Assimp, FreeType2,
                                           and convhull_3d\n7. Recent development focused on UI improvements (vignette effects, post-processing) and documentation updates\n\nThe project appears designed for
                                           exploring relationships between scientific papers in a 3D space with hierarchical clustering visualization. Is this assessment accurate?", content: "Key code
                                           structure:\n- main.cpp: Core rendering loop, handles user input, manages animation state\n- paper_loader.h/cpp: Loads paper data from CSV, manages clustering
                                           hierarchy\n- clusters.h/cpp: Renders cluster convex hulls using Assimp-loaded models\n- opengl/: Contains shader management, mesh/model loading, camera, fonts, etc.\n-
                                           CMakeLists.txt: Linux-focused build (Windows build marked as faulty)\n- Data files: CSV with paper metadata, pre-generated cluster models (.obj), shaders, fonts,
                                           textures")
  ⎿  Yes, your assessment is highly accurate. Based on the file structure and your description, this project is indeed a 3D visualization tool for exploring hierarchically clustered scientific paper data.

     Your breakdown of the components and their functions is spot on:
     … +9 lines (ctrl+r to expand)

● Update Todos
  ⎿  ☒ Explore project structure and identify main components
     ☒ Analyze build system (CMakeLists.txt)
     ☒ Examine main.cpp and core functionality
     ☒ Check with Gemini for understanding verification
     ☒ Review recent commits for project direction
     ☐ Write contributor summary

● This is an OpenGL-based 3D visualization tool for exploring hierarchically clustered scientific paper data from AI research pipelines. The project renders thousands of papers as colored cubes in 3D space, with
  their positions and cluster assignments derived from dimensionality reduction techniques. Papers are grouped into transparent convex hull clusters at multiple hierarchical levels (2-6), allowing researchers to
  explore relationships between papers through interactive navigation with real-time animation that progresses chronologically through the dataset.

  The codebase uses modern C++20 with a CMake build system targeting Linux (Windows support is incomplete). Core dependencies include GLFW3/GLAD for OpenGL context, GLM for 3D math, Assimp for loading cluster
  models, and FreeType2 for text rendering. The architecture cleanly separates concerns: data loading from CSV files, instanced rendering for performance, cluster hull generation/rendering, and UI overlays
  showing statistics like cluster distributions and exploration progress. Recent development has focused on post-processing effects and UI improvements to enhance the visual exploration experience.

● Update Todos
  ⎿  ☒ Explore project structure and identify main components
     ☒ Analyze build system (CMakeLists.txt)
     ☒ Examine main.cpp and core functionality
     ☒ Check with Gemini for understanding verification
     ☒ Write contributor summary
     ☒ Review recent commits for project direction

We can also have Gemini do the initial work and have Claude Code check it:

> Have Gemini analyse this codebase in depth. Critically evaluate the resutls Gemini gives you and make sure they are correct. Produce a short two paragraph summary for a potential new contributor.

● I'll analyze this codebase using Gemini and provide a comprehensive summary for potential contributors.

● Update Todos
  ⎿  ☐ Explore codebase structure and files             
     ☐ Have Gemini analyze the codebase in depth
     ☐ Critically evaluate Gemini's analysis
     ☐ Create two paragraph summary for new contributors

● Update Todos
  ⎿  ☐ Explore codebase structure and files             
     ☐ Have Gemini analyze the codebase in depth
     ☐ Critically evaluate Gemini's analysis
     ☐ Create two paragraph summary for new contributors

● List(.)
  ⎿  Listed 1119 paths (ctrl+r to expand)

● Read(CMakeLists.txt)
  ⎿  Read 92 lines (ctrl+r to expand)

● Read(main.cpp)
  ⎿  Read 557 lines (ctrl+r to expand)

● Search(pattern: "**/*.{cpp,h,hpp,c,txt,md,json}")
  ⎿  Found 100 files (ctrl+r to expand)

● Update Todos
  ⎿  ☒ Explore codebase structure and files             
     ☐ Have Gemini analyze the codebase in depth
     ☐ Critically evaluate Gemini's analysis
     ☐ Create two paragraph summary for new contributors

● Read(README.md)
  ⎿  Read 65 lines (ctrl+r to expand)

● Task(Analyze source files structure)
  ⎿  Done (15 tool uses · 50.0k tokens · 1m 6.4s)                                                                                                                                                                   

● Now I'll have Gemini analyze the codebase comprehensively:

● check_with_gemini:check_with_gemini (MCP)(prompt: "Please provide a comprehensive analysis of this codebase. This is a 3D visualization application for academic papers that uses OpenGL. I want you to analyze
                                           the code quality, architecture, potential issues, and provide insights about its design patterns and implementation approaches. Consider:\n\n1. Code organization and
                                           structure\n2. OpenGL and graphics programming best practices\n3. Memory management and resource handling\n4. Performance considerations\n5. Error handling and
                                           robustness\n6. Code maintainability and extensibility\n7. Any potential security or stability issues\n8. Overall software engineering quality\n\nBe thorough and critical
                                            in your analysis. Point out both strengths and weaknesses.", content: "This codebase is a 3D visualization application called \"AI Pipeline Visualisation\" that renders
                                            academic papers as cubes in 3D space, organized into clusters. Here's the key information:\n\n## Core Architecture:\n- Built with C++20, OpenGL 4.1, CMake build
                                           system\n- Uses GLFW for windowing, GLAD for OpenGL loading, GLM for math, Assimp for 3D models, FreeType for fonts\n- Main components: App (OpenGL wrapper), PaperLoader
                                           (data management), Clusters (3D cluster visualization), Shader management\n\n## Main Files Structure:\n- main.cpp: Main application loop with animation, rendering, and
                                           UI\n- src/opengl/app.cpp: Core OpenGL application framework\n- src/paper_loader.cpp: Loads papers from CSV, generates clusters\n- src/clusters.cpp: Generates and renders
                                            3D convex hulls for clusters\n- src/opengl/shader.cpp: Shader program management\n\n## Key Code Snippets:\n\n### main.cpp (main loop):\n```cpp\n// Main rendering loop
                                           with instanced paper rendering\nwhile (!app.shouldClose())\n{\n    app.handleInput();\n    app.enablePostProcessing();\n    app.clear();\n    \n    // Render papers as
                                           instanced cubes\n    pointShader.use();\n    pointShader.setMat4(\"projection\", app.getPerspectiveMatrix());\n    pointShader.setMat4(\"view\", app.getViewMatrix());\n
                                              glDrawArraysInstanced(GL_TRIANGLES, 0, 36, static_cast<int>(paperData.size() / 5));\n    \n    // Render transparent clusters with depth sorting\n    std::map<float,
                                           std::pair<int, glm::vec3>> sortedClusters{};\n    for (int c = 0; c < std::pow(2, CLUSTER_DEPTH); ++c) {\n        // Sort clusters by distance for proper transparency\n
                                                  distance = glm::length(app.getCameraPosition() - clusterData->position);\n        sortedClusters[distance] = std::make_pair(c, color);\n    }\n    \n    //
                                           Animation and UI updates\n    animationProgress += ANIMATION_SPEED * app.getDeltaTime();\n}\n```\n\n### src/opengl/app.cpp (OpenGL framework):\n```cpp\nclass App {\n
                                           GLFWwindow* window;\n    Camera camera;\n    PostProcessor* postProcessor;\n    \n    void init(int width, int height, const std::string& title) {\n        if
                                           (!glfwInit()) { /* error handling */ }\n        \n        glfwWindowHint(GLFW_CONTEXT_VERSION_MAJOR, 4);\n        glfwWindowHint(GLFW_CONTEXT_VERSION_MINOR, 1);\n
                                            glfwWindowHint(GLFW_OPENGL_PROFILE, GLFW_OPENGL_CORE_PROFILE);\n        \n        window = glfwCreateWindow(width, height, title.c_str(), nullptr, nullptr);\n        if
                                            (!gladLoadGLLoader((GLADloadproc)glfwGetProcAddress)) {\n            /* error handling */\n        }\n    }\n};\n```\n\n### src/paper_loader.cpp (data
                                           management):\n```cpp\nclass PaperLoader {\n    std::vector<Paper> papers;\n    std::map<int, std::map<int, Cluster>> clusters;\n    \n    void loadFromFile(const
                                           std::string& filename, float scale) {\n        // Parse CSV with wide string support for UTF-8\n        std::wifstream file(filename);\n
                                           file.imbue(std::locale(\"en_US.UTF-8\"));\n        \n        // Process each paper line\n        while (std::getline(file, line)) {\n            Paper paper;\n
                                             // Parse coordinates, title, inclusion status\n            paper.x = std::stof(fields[1]) * scale;\n            paper.y = std::stof(fields[2]) * scale;\n
                                           papers.push_back(paper);\n        }\n    }\n    \n    void generateClusters() {\n        // Generate hierarchical clusters (levels 2-6)\n        for (int level = 2;
                                           level <= 6; ++level) {\n            // Group papers into 2^level clusters\n            clusters[level] = generateClusterLevel(level);\n        }\n    }\n};\n```\n\n###
                                           src/clusters.cpp (3D cluster visualization):\n```cpp\nclass ClusterRenderer {\n    std::map<int, std::map<int, ClusterData>> clusterData;\n    \n    void
                                           generateClusters(const std::map<int, std::map<int, Cluster>>& clusters) {\n        for (const auto& [level, levelClusters] : clusters) {\n            for (const auto&
                                           [id, cluster] : levelClusters) {\n                // Generate convex hull from paper positions\n                ConvexHull hull = convhull_3d_build(cluster.vertices);\n
                                                          \n                // Export as .obj file\n                std::ofstream objFile(\"data/cluster_models/cluster_\" + \n
                                           std::to_string(level) + \"_\" + std::to_string(id) + \".obj\");\n                // Write vertices and faces\n            }\n        }\n    }\n    \n    void
                                           renderCluster(const Shader& shader, const glm::mat4& projection, \n                      const glm::mat4& view, const glm::vec3& color, int level, int id) {\n
                                           ClusterData* data = getClusterData(level, id);\n        shader.setVec3(\"color\", color);\n        shader.setMat4(\"model\", data->modelMatrix);\n
                                           glBindVertexArray(data->VAO);\n        glDrawElements(GL_TRIANGLES, data->indexCount, GL_UNSIGNED_INT, 0);\n    }\n};\n```\n\n### Shader
                                           examples:\n```glsl\n//points.vert - Instanced paper rendering\n#version 410 core\nlayout (location = 0) in vec3 aPos;\nlayout (location = 1) in vec3 aNormal;\nlayout
                                           (location = 2) in vec3 aOffset; // per-instance position\nlayout (location = 3) in float aIncluded; // per-instance inclusion flag\n\nuniform mat4 model, view,
                                           projection;\nout vec3 FragPos, Normal;\nout float Included;\n\nvoid main() {\n    vec3 worldPos = aPos + aOffset;\n    gl_Position = projection * view * model *
                                           vec4(worldPos, 1.0);\n    FragPos = worldPos;\n    Normal = aNormal;\n    Included = aIncluded;\n}\n\n//cluster.frag - Transparent cluster rendering\n#version 410
                                           core\nin vec3 FragPos, Normal;\nuniform vec3 color, CameraPos;\n\nvoid main() {\n    // Blinn-Phong lighting\n    vec3 lightDir = normalize(CameraPos - FragPos);\n
                                           float diff = max(dot(Normal, lightDir), 0.0);\n    vec3 result = (0.3 + 0.7 * diff) * color;\n    gl_FragColor = vec4(result, 0.3); // Semi-transparent\n}\n```\n\n##
                                           CMakeLists.txt build configuration:\n```cmake\nproject(main VERSION 0.2 DESCRIPTION \"AI Pipeline Visualisation\" LANGUAGES CXX)\nset(CMAKE_CXX_STANDARD 20)\n\n#
                                           Cross-platform library linking\nif (CMAKE_SYSTEM MATCHES Linux)\n    set(GL_LIBS GL GLU glfw3 assimp freetype)\n
                                           include_directories(${CMAKE_SOURCE_DIR}/include/linux)\nendif()\n\n# Copy assets and shaders to build directory\nadd_custom_target(copy_assets\n    COMMAND
                                           ${CMAKE_COMMAND} -E copy_directory ${CMAKE_CURRENT_LIST_DIR}/data ${CMAKE_CURRENT_BINARY_DIR}/data)\n```\n\nThe application loads academic papers from CSV, renders them
                                           as animated 3D cubes, groups them into hierarchical clusters with convex hull visualization, and provides interactive controls for exploration.")
  ⎿  Error: Gemini request timed out after 30 seconds

● Let me try with a more concise analysis request:

● check_with_gemini:check_with_gemini (MCP)(prompt: "Analyze this 3D visualization codebase for code quality, architecture, and potential issues. Focus on OpenGL best practices, memory management, error handling,
                                            and overall software engineering quality. Provide both strengths and weaknesses.", content: "C++ OpenGL 4.1 academic paper visualization app with these key
                                           components:\n\n**Architecture:**\n- App class: OpenGL/GLFW wrapper, handles window, camera, rendering loop\n- PaperLoader: CSV data loading, cluster generation, UTF-8
                                           support  \n- ClusterRenderer: 3D convex hull generation/rendering for clusters\n- Shader management system\n\n**Key Code Patterns:**\n\nMain loop with instanced
                                           rendering:\n```cpp\nwhile (!app.shouldClose()) {\n    app.handleInput();\n    app.clear();\n    \n    // Instanced paper cubes\n    glDrawArraysInstanced(GL_TRIANGLES,
                                           0, 36, paperCount);\n    \n    // Transparent clusters with depth sorting\n    std::map<float, std::pair<int, glm::vec3>> sortedClusters;\n    for (clusters) {\n
                                           distance = glm::length(cameraPos - clusterPos);\n        sortedClusters[distance] = clusterData;\n    }\n    // Render back-to-front\n}\n```\n\nResource
                                           management:\n```cpp\nclass App {\n    GLFWwindow* window;\n    unsigned int framebuffer, colorTexture, depthTexture;\n    \n    ~App() {\n        glDeleteFramebuffers(1,
                                            &framebuffer);\n        glfwTerminate();\n    }\n};\n```\n\nData loading with wide strings:\n```cpp\nvoid PaperLoader::loadFromFile(const std::string& filename, float
                                           scale) {\n    std::wifstream file(filename);\n    file.imbue(std::locale(\"en_US.UTF-8\"));\n    // Parse CSV data, handle UTF-8 paper titles\n}\n```\n\nOpenGL state
                                           management:\n```cpp\nvoid App::enableDepthTesting() {\n    glEnable(GL_DEPTH_TEST);\n    glDepthFunc(GL_LESS);\n}\n\nvoid App::enableBlending() {\n
                                           glEnable(GL_BLEND);\n    glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA);\n}\n```\n\n**Libraries:** GLFW, GLAD, GLM, Assimp, FreeType, ConvexHull3D\n**Features:**
                                           Instanced rendering, transparent clusters, post-processing, font rendering, CSV data loading, animation system")
  ⎿ First, I'll conduct a comprehensive search of the codebase to identify all C++ and header files. This will give me a complete picture of the project's structure and all its components.Okay, that's a lot of fi
    les. Based on your description and the file list, I'll focus on the core application logic in the `src` directory and the root-level `.cpp` files. The `include` and `lib` directories appear to contain third-p
    arty libraries, so I'll exclude those from the primary analysis.
    … +57 lines (ctrl+r to expand)



● Update Todos
  ⎿  ☒ Explore codebase structure and files             
     ☒ Have Gemini analyze the codebase in depth
     ☐ Critically evaluate Gemini's analysis
     ☐ Create two paragraph summary for new contributors

● Let me critically evaluate Gemini's analysis by examining specific code examples to verify the claims:

● Read(src/opengl/app.h)
  ⎿  Read 183 lines (ctrl+r to expand)

● Read(src/opengl/app.cpp)
  ⎿  Read 100 lines (ctrl+r to expand)

● Search(pattern: "delete|new|malloc|free", include: "*.cpp")
  ⎿  Found 8 files (ctrl+r to expand)

● Update Todos
  ⎿  ☒ Explore codebase structure and files             
     ☒ Have Gemini analyze the codebase in depth
     ☒ Critically evaluate Gemini's analysis
     ☐ Create two paragraph summary for new contributors

● After critically evaluating Gemini's analysis against the actual codebase, I found it to be largely accurate and insightful. The analysis correctly identifies the use of raw pointers (_defaultShader and
  _postProcessor) in app.h:146 and app.h:151, the memory management issue in wstring2string (main.cpp:500-503), and the hardcoded paths throughout the codebase. The assessment of OpenGL best practices, instanced
  rendering usage, and transparency handling is also correct based on the code structure.

● Update Todos
  ⎿  ☒ Explore codebase structure and files             
     ☒ Have Gemini analyze the codebase in depth
     ☒ Critically evaluate Gemini's analysis
     ☒ Create two paragraph summary for new contributors

● Summary for New Contributors

  This is a sophisticated 3D data visualization application built in C++20 with OpenGL 4.1 that renders academic papers as animated cubes in 3D space, organized into hierarchical clusters with convex hull
  visualizations. The codebase demonstrates solid software engineering practices with clear separation of concerns across its main components: the App class manages OpenGL context and rendering pipeline,
  PaperLoader handles CSV data ingestion with UTF-8 support and cluster generation, ClusterRenderer creates and renders 3D convex hulls using the ConvexHull3D library, and a comprehensive shader management
  system. The application uses modern OpenGL techniques including instanced rendering for performance, proper transparency handling with depth sorting, and post-processing effects, while supporting real-time
  animation controls and interactive 3D navigation.

  The architecture is well-organized with appropriate use of modern C++ features, RAII principles for resource management, and established graphics programming patterns, though it has some areas for improvement
  including inconsistent error handling, occasional use of raw pointers instead of smart pointers, and hardcoded configuration values. New contributors should be comfortable with OpenGL/graphics programming
  concepts, understand 3D mathematics (GLM library), and be familiar with build systems like CMake. The codebase includes comprehensive asset handling (3D models, shaders, fonts, textures), supports
  cross-platform development (Linux focus with Windows compatibility), and implements a complete data pipeline from CSV loading through hierarchical clustering to real-time 3D visualization with interactive
  controls for exploring academic paper datasets.

As you can see one of the downsides is that some of the calls to check_with_gemini take longer than 30 seconds and so timeout.

Qwen3 Leads the Pack: Evaluating how Local LLMs tackle First Year CS OCaml exercises

2025-05-06T17:52:00+01:00

TL;DR Qwen3's self-hostable sized models are very strong and would score top-marks on Cambridge's first year Computer Science OCaml exercises.

Large language models (LLMs) are transforming software development. What if you use a language outside the mainstream that has relatively little training data though? Worse, what if you want to use features that are still under development or only recently released? The model can't have seen those features during training and so won't make use of them when generating unless explicitly prompted to do so. My colleague Jon Ludlam, Anil Madhavapeddy and I have been thinking about this in the context of OCaml and some of the new extensions under development at Jane Street. Having a process for teaching models to use novel language features would not just be useful for interactive code generation but could also enable retrofitting existing codebases or if cheap enough, could be used in the design process for the features themselves.

Before we dived in though we wanted to understand just how good the kind of models are that a developer could run either locally or on a shared local inference server? To do that we used the OCaml tutorial problems first year Computer Science students at the university tackle on the Foundations of Computer Science course taught by Anil and Jon. These are interactive jupyter notebooks where students populate answer cells and their solutions are automatically checked. Each tutorial is known as a 'tick' and has one or more questions. Students must complete earlier questions in order to progress to later questions. There are also 'starred' ticks which are essentially stretch goals and harder than the non-starred ones.

Terminology

A quick bit of terminology that will help in the later sections:

Tokens are the units that Large Language Models take as input and output. A token is roughly three quarters of a word in length on average and they come from a fixed vocabulary for a given model. Here's Deepseek R1's vocab (warning it's about 8mb of json).

Reasoning models or a thinking mode when the model is given some space in which to reason through its approach to the problem before it generates an answer. After pre-training on very large text corpora reasoning models undergo a phase of Reinforcement Learning. If you want to know more, Sebastian Raschka has a great overview article. The latest Qwen3 models from Alibaba have the ability to selectively enable thinking mode. This lets you trade off latency and performance. More on this later.

OpenRouter is a marketplace for LLM providers. It provides a unified OpenAI-compatible API and then routes requests to different model providers based on the model you specify and any filtering options (e.g., no training on data, support for certain features). If you only use proprietary models then this is less interesting, they are often only available via a small handful of APIs that largely have similar pricing (e.g., for Gemini you can go via the Gemini or Vertex API). If you use open weights models though, there are a plethora of providers all competing on price, latency and throughput. For example if you want Llama 3.3 70B you can currently choose the cheapest (inference.net) at $0.10 per million tokens input and $0.25 per million tokens output with 1.02s time-to-first-token and 28.41 tokens per second throughput. On the other hand, if throughput is your primary concern then there's Groq at 372.2 tokens per second but you'll pay a premium at $0.59 in and $0.79 out.

Distillation is when you use the output of a much larger model to improve the performance of a smaller model. When releasing Deepseek-R1, Deepseek also released a series of smaller (originally non-Deepseek) models that had been fine-tuned on the outputs of Deepseek-R1.

Mixture of experts is a type of architecture where only a subset of the network weights are activated for each token processed. This can lead to a performance win at inference time and is in contrast to a dense model in which every process token interacts with every non-embedding model parameter.

Models

We took the following open weight models, using instruct variants models where available:

Google Gemma3 12B and 27B
Deepseek-R1-Llama 70B
Meta Llama-3.1 8B and 70B
Microsoft Phi-4
Mistral 8B
Mistral Nemo (12B)
Mistral Small 24B
Qwen 2.5 7B and 72B
Qwen 2.5 Coder 7B and 32B
Qwen 3 8B, 14B, 32B and 30BA3B (Thinking and non-thinking)

We gave models a similar setup to students. Each model had 3 attempts to complete each question by producing a code block that would pass the notebook's tests. If they produced a passing block of code they proceeded to the next question. We repeated this 5 times per model per tick. We used OpenRouter for inference.

The tick questions tested a range of concepts such as recursion, data structures, streams etc.. We'll look at them in more depth in the results.

Results

Here's a graph of the success rate against parameter count for all models. Note the logarithmic x-axis:

And a table of overall success rate. This is the percentage of tick-question pairs that a model was able to solve given 3 attempts.

Model	Overall Success Rate
anthropic/claude-3.7-sonnet:thinking	96.4%
qwen/qwen3-32b:thinking	95.2%
qwen/qwen3-14b:thinking	84.2%
qwen/qwen3-30b-a3b:thinking	83.0%
qwen/qwen-2.5-coder-32b-instruct	77.6%
meta-llama/llama-3.3-70b-instruct	69.1%
qwen/qwen3-8b:thinking	66.9%
qwen/qwen3-32b	62.4%
mistralai/mistral-small-3.1-24b-instruct	58.2%
qwen/qwen3-14b	56.4%
qwen/qwen3-30b-a3b	48.5%
deepseek/deepseek-r1-distill-llama-70b	44.2%
google/gemma-3-27b-it	33.9%
qwen/qwen3-8b	33.9%
microsoft/phi-4	32.1%
google/gemma-3-12b-it	28.5%
mistralai/mistral-nemo	26.1%
meta-llama/llama-3.1-8b-instruct	25.5%
mistral/ministral-8b	15.8%

High-level takeaways

Qwen 3 models performed very well The best performing model was qwen3-32b in thinking mode at 95.2%. This is very close to Anthropic's Claude 3.7 Sonnet with thinking. At Qwen3 32B's current $0.10 per million input tokens and $0.30 per million output tokens this is up to 50x cheaper than Claude 3.7's $3 M in / $15 M out pricing. Even better the Qwen3 32B model is Apache2 licensed and so can be self-hosted. In addition, all Qwen3 models in non-thinking mode outperformed comparably-sized models from other families.

Qwen2.5's coding variant was the best non-thinking model. The Qwen2.5-Coder-32B model performed well at 77.6% and was the best non-thinking model. It outperformed Qwen3 32B and this means the upcoming Qwen3-Coder model should be very strong.

Qwen3 with thinking on performed better than non-thinking variants. In every case allowing Qwen3 to think significantly improved performance. The qwen3-8b model with thinking on was very close in performance to llama-3.3-70b, a model almost an order of magnitude larger. Note though that this comes at the expense of increased latency and cost. In most cases thinking added 2-3,000 tokens of reasoning before the model produced its final answer, for Qwen3 32B on openrouter that is roughly a minute of thinking. There were also cases where the model entered a loop and kept reasoning until it hit the token limit.

The Qwen3 30B-A3B MoE model could be a good compromise Whilst it has 30 billion parameters the MoE model only activates about 3 billion of them for each token processed. This can be a substantial performance win when doing inference and indeed you can see the throughput ratio on OpenRouter between Qwen3 32B and Qwen3 30B-A3B. I suspect that gap will widen as providers do more optimisation of their MoE serving code.

Within a family more parameters means better performance. Within the Gemma and Qwen families, larger parameter counts generally led to better performance, as seen in the graph. Presumably there is some OCaml in the pretraining corpus and the larger models have a higher chance of retaining information from it?

Deepseek R1's distilled Llama 3.3 70B reasoning model did not beat base model. The Deepseek-R1-Llama-3.3-70B model which was produced by Deepseek by distilling from their large Deepseek R1 model had worse performance than the model it was based on Llama-3.3-70B.

What went wrong?

There were some consistent themes amongst failures across especially the smaller models. Here are some of the most common:

Syntax errors

Many models had basic syntax errors in their solutions, such as incorrect comment syntax or a lack of in. There were also cases of incorrect function application syntax. A very common mistake was not including rec for recursive functions.

Type system confusion

On several of the tick questions this involved confusing the int and float operators (+/+., /.). A common pattern was changing just one of the operators in response to a compile error, then changing another on the next error, etc..

Hallucinated functions

Some failures came from hallucinating functions such as List.sub or List.combinations, as well as assuming things like Core and Format were available. This is curious as even small models (like Qwen2.5-Coder-7B) assumed Core was available and yet still made fundamental OCaml syntactic errors.

Recursion

Beyond the failure to include rec the models struggled with recursion. There were several incorrect base cases in questions involving streams which resulted in a fair bit of non-termination in those questions. It would be interesting to try rewriting some of the tick questions as iterative and seeing whether success rates increase.

Limitations

These are only introductory tasks aimed at first year Computer Science students and so are not representative of the complexity of larger OCaml codebases that may also use advanced language features.

Summary

Qwen3 had an impressive showing. I had actually already written a draft of this blog post before Qwen3 was released, and re-running the evaluation to include it led to a very different outcome!

So what next? It would be interesting to understand where the gap in performance is when thinking mode is on and off. Are there common mistakes Qwen3 is making when thinking is off?

As a next step we clearly need to move beyond first-year Computer Science questions, especially as the top models are already saturating on this test. Anil reports performance on OCaml vibe-coded projects through Claude lags behind its abilities in Python.

Next, how do we improve their performance? For this we really need more data. We now have a dataset of OCaml code extracted from opam that is regularly updated but still small (~80k ml/mli files) and it is unclear as to the difficulty level of each bit of code. This is important because there is good evidence (Sun et al, Ji et al) that going from easy to difficult examples when training via reinforcement learning can improve performance.

One promising avenue is Cassano et al who have done some work leveraging training data from 'high resource' languages to generate code and validation tests in target 'low resource' languages. This approach could be extended to novel language features, as well as being used for reinforcement learning rather than just fine-tuning.

Unrelated to OCaml, I'm looking forward to testing out how these models perform on the Conservation Evidence work that Anil and I are involved in. We currently self-host Deepseek-R1-Llama-3.3-70B and switching to Qwen3 32B could substantially improve the performance at one of our crucial bottlenecks.

A dataset of OCaml's Opam

2025-05-01T10:01:00+01:00

If you have tried to use locally-hostable language models to develop OCaml code, then you will have noticed their performance significantly lags in more niche languages compared to Python or Javascript. Jon Ludlam, Anil Madhavapeddy and I have been doing some work on this recently and there will be more on that soon.

To improve code models, we first need data. To help with that I've created opam-archive-dataset which periodically takes the code for all packages from the ocaml/opam:archive docker image, filters for the most recent version of each package, and then converts everything into the columnar parquet format. This is a very efficient format and results in a ~800MB set of files.

To use the dataset and run queries over it, you can use the Hugging Face datasets library or if you prefer SQL then you can do the following:

# clone the dataset from huggingface
sadiq@server:opam-archive$ git clone https://huggingface.co/datasets/sadiqj/opam-archive-dataset
Cloning into 'opam-archive-dataset'...
remote: Enumerating objects: 17, done.
remote: Total 17 (delta 0), reused 0 (delta 0), pack-reused 17 (from 1)
Unpacking objects: 100% (17/17), 4.31 KiB | 315.00 KiB/s, done.
Filtering content: 100% (3/3), 388.79 MiB | 14.30 MiB/s, done.

# grab clickhouse
sadiq@server:opam-archive$ curl https://clickhouse.com/ | sh
Successfully downloaded the ClickHouse binary, you can run it as:
    ./clickhouse

You can also install it:
sudo ./clickhouse install

# we do not need to install it! We use clickhouse local
sadiq@server:opam-archive$ ./clickhouse local

./clickhouse local
ClickHouse local version 25.5.1.1804 (official build).

:) -- let's have a look at a few rows
SELECT * FROM file('opam-archive-dataset/data/', Parquet) LIMIT 1;

Query id: 0f786705-1568-40ac-837b-004457c3519d

Row 1:
──────
package_name:  dune-action-plugin
version:       3.18.1
license:       MIT
homepage:      https://github.com/ocaml/dune
dev_repo:      git+https://github.com/ocaml/dune.git
file_type:     dune
file_path:     dune-3.18.1/test/blackbox-tests/test-cases/formatting/feature.t/enabled/dune-ocaml-syntax/dune
file_contents: (* -*- tuareg -*- *)

let
() =
Jbuild_plugin.V1.send {|
(alias
 (name runtest)
 (action (echo "ocaml syntax")))
|}

:) -- Let's count how many rows we have
SELECT COUNT(*) FROM file('opam-archive-dataset/data/', Parquet);

SELECT COUNT(*)
FROM file('opam-archive-dataset/data/', Parquet)

Query id: 3ee6eb4b-13b7-47aa-be67-d027c81b47b0

   ┌─COUNT()─┐
1. │  198862 │
   └─────────┘

1 row in set. Elapsed: 0.013 sec. 

:) -- How many unique packages are spawning Domains?
SELECT COUNT(DISTINCT package_name) FROM file('opam-archive-dataset/data/', Parquet) WHERE position('Domain.spawn', file_contents) > 0;

SELECT COUNTDistinct(package_name)
FROM file('opam-archive-dataset/data/', Parquet)
WHERE position('Domain.spawn', file_contents) > 0

Query id: 6f0978d9-3907-4572-bf5e-99aa4e2fceb8

   ┌─COUNTDistinct(package_name)─┐
1. │                         193 │
   └─────────────────────────────┘

1 row in set. Elapsed: 0.723 sec. Processed 197.86 thousand rows, 402.85 MB (273.81 thousand rows/s., 557.48 MB/s.)
Peak memory usage: 385.88 MiB.

We currently extract the package name, version, license, dev repo, file type (dune, opam, mli, ml, .c and .h), file path and the contents itself.

If there are any extra fields that would be useful, let me know. Enjoy!

JSON output from Deepseek R1 and distills with llama.cpp

2025-01-30T17:52:00+00:00

We're evaluating Deepseek R1 and its distills for our project on identifying and extracting evidence from literature. One part of our pipeline needs a strong reasoning model with JSON structured output.

Using llama-server's OpenAI compatible completion endpoint with response_format and JSON schema conflicts with the model's reasoning in <think> tags. Here's a workaround using the grammar functionality.

First, grab json_schema_to_grammar from the llama.cpp repo.

Next you can use the following:

def convert_schema_to_grammar(json_schema: dict) -> str:
    converter = json_schema_to_grammar.SchemaConverter(
        prop_order={},
        allow_fetch=False,
        dotall=False,
        raw_pattern=False
    )

    converter.visit(json_schema, 'json-schema')
    json_grammar = converter.format_grammar()

    # do the root gbnf grammar that handles the <think> and </think> tags
    # lack of less-than didn't hinder performance for us but ymmv!
    base_rules = """
root ::= "<think>" [^<]+ "</think>" [\\n]* json-schema
"""

    return base_rules + json_grammar

Then use the 'extra_body' parameter in your request:

client = OpenAI()

params = {
    "model": "...",
    "extra_body": {
        "grammar": grammar
    }
}

response = client.chat.completions.create(
    **params
)

Example below:

Provide a fictional name and address using the following JSON schema:

{
  "type": "object",
  "properties": {
    "name": {
      "type": "string"
    },
    "age": {
      "type": "integer"
    },
    "address": {
      "type": "object",
      "properties": {
        "street": {
          "type": "string"
        },
        "city": {
          "type": "string"
        },
        "zip": {
          "type": "string"
        }
      }
    }
  }
}

<think>
Okay, so the user wants me to provide a fictional name and address following a specific JSON schema. Let me break this down.

First, the schema has an object with name, age, and address. The address itself is another object containing street, city, and zip. I need to make sure each field is of the correct type: strings for name, street, city, zip, and an integer for age.

I should come up with a plausible name. Maybe something common like Emily Carter. Age should be an adult, say 32. For the address, I'll pick a street name, perhaps Maple Street, a number like 147. The city could be something like Riverton, and the zip code needs to be a 5-digit number, maybe 12345.

Wait, is Riverton a real city? I think there are places named Riverton in various states, but since it's fictional, it doesn't matter. Alternatively, I could make up a city name, but using a real one might be easier.

Putting it all together, I'll structure the JSON with these values. I need to ensure that the syntax is correct, with proper commas and brackets. Also, the keys should match exactly: name, age, address, street, city, zip.

I should double-check that all the types are correct. Name is a string, age is an integer, and each part of the address is a string. That should satisfy the schema requirements.

I think that's all. Time to put it together in the response.
</think>

{
  "name": "Emily Carter",
  "age": 32,
  "address": {
    "street": "147 Maple Street",
    "city": "Riverton",
    "zip": "12345"
  }
}

This approach lets you preserve model reasoning while enforcing JSON output structure. Enjoy!

Teaching Bloom Filters new tricks

2020-12-25T00:52:00+00:00

I've had a few interesting conversations around text indexing recently with friends and something that's fallen out is that there are some really cool data structures you build that are extensions of Bloom filters. The techniques for constructing them don't seem to be that widely known so this blog post intends to remedy that.

We will start by examining a variant of the traditional Bloom filter and then we'll look at ways we can generalise the model to produce new types of probabilistic data structures and understand what properties they might have.

Bloom Filters

To begin with, let's take a look at what a Bloom filter is and how it works so we're all on the same page.

Loosely a Bloom filter is a data structure into which you can add a set of elements. Later you can query it with any of those elements and it will return true. Crucially it might also return true for elements that weren't in the set with some (tunable) probability. The benefit you get from this trade-off is that Bloom filters can be very compact.

Trade-offs

How big this benefit is depends on two things. First, how high a probability of an incorrect answer (a false positive) your application can tolerate and second, how large the items are in the set you want to track.

For the first case, let's look at a simple example. Imagine if you were a social network and you want to maintain a cache of your most active ten million users. Each user has is identified by a 32-bit integer. If we naively stored the identifiers of our most active million users we're need 320 million bits or 40 megabytes. On the other hand, if we're willing to accept a 1% false positive rate then a Bloom filter needs just over 11 megabytes. A decent win.

The benefits become much greater as soon as you have large keys. As we'll see in a minute this is because Bloom filters use hashing to avoid storing the keys themselves. If instead of 32-bit user identifiers we are storing urls which could be 320 bits upwards then the win becomes significantly greater. Storing ten million is suddenly >400 megabytes but a bloom filter still clocks in at 11 megabytes for a 1% false positive rate or 17 megabytes for a 0.1% false positive rate.

False positives

Bloom filters are suited to situations where they can filter out the need to do some work and where a false positive just means some wasted work. A good example would be a local Bloom filter sitting in front of a remote key-value cache. The filter can contain the set of keys the remote cache holds the values for and can filter out needless network requests for keys that aren't present. A false positive in this case means we end up doing a request to the remote cache when not necessary.

How they work

Let's look at how a Bloom filter actually works. For those already familiar with Bloom filters, for presentation purposes we're only going to discuss Partitioned Bloom filters as this makes the mathematics simpler and exact.

A Bloom filter consists of an array of $m$ bits $B$ and $k$ hash functions $h_0..h_{k-1}$. The hash functions take a key $x$ and map it to a location in $B$. The hash functions are assumed to be independent of each other. That is if we hash some value "foo" with $h_0$ then the result should give us no information about the result from hashing with $h_1$.

Adding to a filter

To add to the Bloom filter we hash the key with the hash functions and use their results to set bits in the array $B$. More specifically:

for i = 0 to k-1
    j = h[i](x)
    B[i*k+j] = true

Each hash function indexes into a subrange of size $\frac{m}{k}$ i.e if there are 4 hash functions and array $B$ is 256 bits then the first hash function picks a position in 0 to 63, the second hash function 64 to 127 and so on. These are later referred to as a hash function's partition.

Querying a filter

To check a Bloom filter for an element we again use the hash functions to hash the key (resulting in the same positions we would have had if we had added the element) and check the array positions they generate in $B$. Again, some some pseudo code:

p = true /* whether the element is present or not */
for i = 0 to k-1
    j = h[i](x)
    p = p & B[i*k+j]
return p

Why they work

Let's reason out how this works. If we have the element "foo" and add it to a Bloom filter, it will result in a number ($k$) of positions in the array $B$ being set to true. If we then test that same Bloom filter with "foo" we check those same positions and if they are all true then we probably added "foo" at some point in the past to the filter. There can never be a false negative, if we added "foo" those bits must have been set.

There can however be false positives, we may have added some other keys which just so happened to generate positions that covered all the ones that "foo" would hash to. This is the source of false positives in Bloom filters. We can actually calculate the probability that this will occur.

False positive probability for k=1

To simplify let us have a Bloom filter with a single hash function (so $k=1$) and with array $B$ of size $m$ bits.

A single element is added with the routine we specified above. What is the probability that a bit in $B$ is set? Since $h_{0}$ can pick any position¹ then the probability that a bit is set is $\frac{1}{m}$.

Now comes testing. We test the single element we added. The position will be the same as when we added it, so we'll correctly return that it is present. What if we tested a different element to the one we just added to the Bloom filter? What is the probability that we get a false positive? For a false positive to occur the element we are testing must map to a position that was already set in the array $B$. We've already calculated that as $\frac{1}{m}$.

Now instead of a single element, let us consider what happens if we add $n$ elements and then test something not in those elements.

For each element we add, the probability that a bit position won't be set is $1-\frac{1}{m}$. After adding $n$ elements, the probability the bit hasn't been set is $(1 - \frac{1}{m})^n$.

Testing an element that wasn't one we added, we want to know the probability that the bit has been set by one of the previous $n$ elements. This is complement of the event that it hasn't been set previously, so:

$$1 - (1 - \frac{1}{m})^n$$

An intuitive way of looking at this is that it is the percentage of 1s set in $B$ after we add $n$ items. To test out our simple filter, if the size of $B$ ($m$) is 512 bits and we add 32 elements then the false positive rate if we test a new element that wasn't in the original set is:

$$1 - (1 - \frac{1}{512})^{32} = 0.061$$

So about 6.1%. If we add another 32 elements this increases to 11.8% and so on.

Generalising to multiple hash functions

How do we extend this to multiple hash functions? For partitioned Bloom filters we don't need to deal with collisions between hash functions e.g where $h_0("foo")$ and $h_1("foo")$ give the same result. To reiterate, a false positive is where all bit positions in $B$ for an element tested, but not initially added, are set.

Let's add a second hash function to our simple filter. Now we have $h_0$ and $h_1$. To get a false positive we would have needed the bit in $h_0$'s partition to be set and the bit in $h_1$'s partition. The probability that a bit is set in the 0th partition when an element is added is $\frac{1}{\frac{m}{2}} = \frac{2}{m}$ because each partition is now $m/2$ in size as there are two) That the bit is not set after $n$ elements are added is $(1 - \frac{2}{m})^n$. For the two (independent) hash functions we end up with:

$$(1 - (1 - \frac{2}{m})^n)^2$$

This generalises to:

$$(1 - (1 - \frac{k}{m})^n)^k$$

We can quickly work our earlier example again. If B is 512 bits ($m = 512$), have 4 hash functions ($k = 4$) and we add 32 elements ($n = 32$) then we end up with:

$$(1 - (1 - \frac{4}{512})^{32})^4 = $$

Which is 0.2%. Increasing k initially reduces the false positive rate but there's an optimum number ($k_{opt}$) before the rate starts to rise. In this example that is $k = 11$:

Generalising Bloom filters

Now we understand how a Bloom filter works, let's look at how we can generalise the technique and apply it to create some data structures with interesting properties. We'll start by sketching out a general structure and then look at how traditional Bloom filters fit within it.

For the generalised data structure we have an array $B$ as before but instead of bits we have state of type $T$ i.e $B[0]$ is data of type T at the first position in array $B$.

To add to the generalised Bloom filter we have an input element $x$ and associated data $y$. We use $k$ hash functions ($h_0$, $h_1$, etc..) as before. The algorithm is:

for i = 0 to k-1
    j = h[i](x)
    B[i*k+j] = combine(B[i*k+j], y)

So the main difference from before is that we use the combine function and element's associated data to update the state at the hash partition location.

Now let's look at how we would query the generalised Bloom filter:

p = list() /* empty list */

for i = 0 to k-1
    j = h[i](x)
    p.append(B[i*k+j])

return reduce(p)

Here we take the state each hash partition location, make a list of them and pass them to a reduce function. The reduce function returns a result that is some value $d$ or none.

Lastly we're going to state that the result from querying our generalised Bloom filter will have a one-sided error. We define a third function error and say that if an element $x$ with associated data $y$ is added to the generalised Bloom filter then querying for element $x$, will always contain some value $d$ and the function error($y$, $d$) will always return true. If an element $x$ with associated data $y$ was not added to the filter then querying will return none or with some (tunable) probability a value $d$ which error($y$, $d$) could return true or false for.

So to summarise, our parameters are a type $T$ for the data array and three functions combine, reduce and error. It turns out that if we pick those four parameters carefully, we can maintain a one-sided error. We'll start with the traditional Bloom filter and then get more adventurous.

Fitting the traditional filter

$T$ in the traditional filter is a single bit. combine is the max function², reduce is the min function and error is equals.

We also need to modify the input slightly, $x$ is still the element to be added but the associated data $y$ is 1 if the element is in the set we added to the filter and 0 if it is not.

If you plug these in to the earlier pieces of pseudo code you should be able to convince yourself that the filter performs identically to the standard description of a partitioned Bloom filter.

Maxmin-based Bloom filters

It turns out if we leave combine as the max function, reduce as the min function and error is $d \geq y$ then we can still get one-sided errors as long as the elements of type T have a total order.

By a total order we mean that we can compare any two elements of type T³. We'll see later why we're being very specific about this.

Here's an example filter that would satisfy the above requirements. Assuming the associated data $y$ for each element is a positive integer (think files with a corresponding sizes in kilobytes) then we could use a maxmin-based Bloom filter with T being an unsigned integer. We initialise all elements of $B$ to 0 and don't allow 0 as a $y$.

To add to the example maxmin filter:

for i = 0 to k-1
    j = h[i](x)
    B[i*k+j] = max(B[i*k+j], y)

And to query:

p = list() /* empty list */

for i = 0 to k-1
    j = h[i](x)
    p.append(B[i*k+j])

return min(*p) /* min in list */

For an $x$ added to the example filter, a query returns a result greater than or equal to the associated data $y$ seen with $x$. For $x$ that wasn't added to the filter we will return none or with some probability, a random number from the filter.

There's actually two types of error here:

We test an element $x$ that we haven't added to the Bloom filter and get a value back for it instead of none. We call this a false positive.
We test an element $x$ that we have added to the Bloom filter and get a value back that is higher than the associated value $y$ we originally added.

Let's start with the false positive probability. We can reuse most of the reasoning from the partitioned Bloom filter false positive derivation earlier. Instead of the bit in $B$ having been set by a different $x$ we're instead interested in the element in $B$ being greater than zero, which means it has been set before.

Since we have $m$ elements in $B$, the probability that an element is set to greater than zero on adding to the filter is still $\frac{1}{m}$. After $n$ additions the probability that it is still zero is $(1 - \frac{1}{m})^n$. So we again have the same probability of giving a false positive as the traditional case:

$$(1 - (1 - \frac{k}{m})^n)^k$$

What about the second error? For this to happen each of the $k$ hash partition locations for an element $x$ must contain a value that is higher than the associated data $y$. They all must because we take the minimum. There's a bit of difficulty involved in calculating this directly since this kind of error requires a hash collision in each partition and for that collision to result in a higher resulting value. The probability clearly depends on $y$, if $y$ is very low (or even the minimum we could see, which is 1) then any collision may lead to this error. If $y$ is the maximum we could see then no collision will lead to this error.

For now let us go with a very pessimistic assumption that a collision results in a higher value, implying $y$ is the minimum we will see. This gives us an upper bound on the probability of the second type of error.

Assume we add $x$ to the filter and we now add a second item. Again we start with a single hash function $k = 1$ and element array $B$ of size $m$. What is the probability there was a collision? We picked a location randomly (using $h_0(x)$) for our element $x$ and now we're picking one again for the second one - the probability of picking the same one is $\frac{1}{m}$.

Now we add $n$ items, what's the chance we didn't collide? Not colliding after one is $1-\frac{1}{m}$. After $n$ items we end up with $(1-\frac{1}{m})^n$. Similar to earlier on with the traditional Bloom filter but reversed, an intuitive way of looking at this is the percentage of unset elements in $B$.

So for a single hash function, our estimate ($d$) for the associated data $y$ is exact with probability at least $(1-\frac{1}{m})^n$. Specifically:

$$p(d-y > 0) \leq (1-\frac{1}{m})^n$$

What is the situation with multiple hash functions? As we are taking the min then we only need one of the partitions not to have a collision to get an exact answer e.g not where there has been a collision in every partition.

The probability of having a at least one collision in a partition after $n$ elements is:

$$1 - (1 - \frac{k}{m})^n$$

for $k$ partitions this is:

$$p(d - y > 0) \leq (1 - (1 - \frac{k}{m})^n)^k$$

Which is the same as the equation we worked out for the first class of error! If we step back though this should make intuitive sense. Both class of error depend on the proportion of data elements that are non-zero in $B$. In the first class of error we might read all non-zero elements and incorrectly return a value instead of none and in the second class of error the more set elements we have the greater the chance we have of colliding in every partition.

I should point out at this point that this is a very pessimistic bound. It assumes that any collision leads to an increase in the stored value. There's also a question we've skirted around in this discussion and that is if we make an error of the second class, how big is our over-estimate? I think there's a follow-up blog post on that - though at this stage I'm not sure we can make statements on the over-estimate without making assumptions as to the distribution of $y$. I may be wrong, let's discuss on twitter: tweet me.

Another thing to note about this filter is that while we only specified it for elements $x$ with accompanying data $y$, you can add $x$ multiple times to the filter with different $y$s and the filter will return an estimate for the maximum $y$ it encountered.

Generalising this model further

Just to recap. For Maxmin filters we have combine as the max function, reduce as the min function and error is $d \geq y$. Our elements are of type T and have a total order.

We can go further than this with a technique Boldi & Vigna showed in 2004 called Compressed Approximators.

For Compressed Approximators we have combine as the least upper bound, reduce as the greatest lower bound and error is still $d \geq y$. We'll define these terms in a second. The elements are of type T and form a lattice. For T, this means two things.

First, it means we have some way of comparing two elements of type T, a partial order. The partial part means not all elements are comparable - this is contrast to a total order, where all elements are.

The second is that for any two elements of type T we can find:

the smallest element of type T that is greater than or equal to those two elements (the least upper bound)
the largest element of type T that is less than or equal to those two elements (the greatest lower bound)

This is convenient because we need to be able to do those two things for combine and reduce.

We'll look at an example now to try to build up an intuitive feeling of what's going on. We base our example around the inclusion order of sets, which is a partial order⁴. Inclusion order means that if all elements of A are also in B i.e A is a subset of B ($A \subseteq B$) then $A \leq B$. For inclusion order, the least upper bound of two sets is their union and the greatest lower bound is their intersection⁵.

A multi-member Bloom Filter

Armed with this lattice, let's turn to a practical application. Imagine you have assets with a key $x$ that may be cached on one or more of multiple caching servers all of which are infront of some authoritative source (an origin for a CDN, for example). The caching servers may be ephemeral and so we don't want to simply rely on hashing the key to a single server.

The most naive option would be to maintain a Bloom filter for each of the servers but this requires us to potentially do as many tests as we have servers. We can construct a Compressed Approximator for this problem which we only need to test once.

To add to the multi-member Bloom filter we do the following, assuming $x$ is our key, $y$ is the server it is now cached on, $k$ is the number of hashes and $B$ is an array of sets.

for i = 0 to k-1
    j = h[i](x)
    B[i*k+j] = B[i*k+j].add(y)

So for each hash function and partition, we simply add $y$ to the set at the hash's index. Assuming add adds to the set if it exists or if not, instantiates it to the singleton set containing just the parameter.

To query the filter we do:

p = set() /* empty set */

for i = 0 to k-1
    j = h[i](x)
    if i == 0 then
        p = B[i*k+j]
    else
        p = intersection(p, B[i*k+j])

return p

We loop over each hash function and partition, taking the intersection of all the sets at those locations⁶.

Again we have a data structure with two possible types of error. As with the maxmin filter, the first type of error is that we test an element we have not added to the filter and we will mistakenly return a set of servers it may be cached on instead of the empty set.

The second type of error is if we test an element we have added to the filter we may get back a set of caches that contains servers the item is not cached on in addition to the right server.

We can use the derivation from earlier for the first type of error, the probability of a false positive is:

$$(1 - (1 - \frac{k}{m})^n)^k$$

For the second error we can also make a similar argument to the maxmin filter, our probability of getting an extra caching server back is at most that of our false positive rate⁷.

Recap and conclusion

So to recap, we've discussed the standard Bloom filter and derived a probability for the false positive rate of the partitioned variant. We then looked at maxmin Bloom filters which generalise them to elements that have a total order, including deriving probabilities for the false positive rate and the event that the value the filter returns is higher than the original value added. Finally we looked at a further generalisation for elements that have a partial order, Compact Approximators, and looked at one specific data structure using that model.

We should be able to use this knowledge to come up with other novel probabilistic data structures based on Bloom filters that have properties which make them better suited to practical problems. I have a whole laundry list of other ones to write up for the next holiday. Want to stay tuned? Follow me on twitter: @sadiqj

Hopefully with uniform probability, if it's a good hash function. ↩
Technically it doesn't have to be the max function, you're effectively ignoring the existing state but the symmetry with min as reduce makes the narrative a bit easier later on. ↩
There are further restrictions on how the comparison must work, see the wikipedia page for more information. ↩
We're just going to casually steam past this but Wikipedia has a page with the formal definitions. We're also going to handwave a bit around the set definitions in this bit to keep things short. The take aways should be how to use the framework to develop new data structures. ↩
This only works if all the elements that could be in either set come from some other set, say S. More specifically we say that both of the sets belong to the power set of S. ↩
We could optimise this by early-outing if we ever get an empty set from a hash location. ↩
We can probably make the bound stronger by making some assumptions on how we use the filter and how many caching servers we add items to. That's material for another post though! ↩

Getting OCaml running on the ESP32

2017-01-15T00:00:00+00:00

I was looking for some small Christmas stocking-fillers to give to techie friends and decided to try to find some interesting electronics boards from China.

In the end, I went with the WEMOS Lolin32 Lite which features Espressif's ESP32. If you're not familiar with the ESP32, it's an awesome little chip that features the following:

Dual core 240Mhz 32-bit Xtensa LX6s
Wi-Fi (802.11 b/g/n) and Bluetooth (v4.2 + BLE)
520kb of SRAM
A separate ultra-low power processor

The Lolin32 Lite couples that with 4mb of flash, micro-usb connection and Li-Po charging circuitry.

Espressif has a gcc-based toolchain and an "IoT Development Framework" which provides a port of Newlib, FreeRTOS, LWIP and a whole host of other frameworks.

My friend Anil suggested that with a gcc and libc, porting of the OCaml interpreter would be fairly easy. He was mostly right.

Caml runtime

It took a little while to understand how OCaml's build system worked and thankfully it seems there's been good support for cross compilation since 4.02. The configure script has a pretty funky way of determining features of the compiler and runtime by compiling lots of small C programs and seeing what builds. This required a few small changes where features were detected but only partially available via Espressif's port of Newlib. Posix signals and BSD sockets were two cases where this happened. I should expand the code for the tests to cover the missing functionality and try to upstream it, which would avoid the configure script hacks.

Rebuilding Newlib

Unfortunately either I was failing or Espressif's build of newlib doesn't seem to include signal(). This meant a rebuild of newlib was required without the SIGNAL_PROVIDED flag, which includes an implementation of signal(). There's also no support for directories in the IDF, so I had to stub out some parts of sys.c and unix.c. With those changes it was possible to get libcamlrun.a compiled.

Once I had a cross compiled bytecode runtime, I was most of the way there.

Building an image

Next step was to get some OCaml compiled which could then be incorporated into the image to flash. This is actually pretty simple with ocamlc and custom runtimes and you end up with a C source file you can then throw in to the rest of the IDF component build system. I wrote a little bit of C that kickstarted ocaml via caml_startup and had a buildable image to flash.

Debugging

I flashed the board and immediately got an abort after malloc failed. First things to tweak were the garbage collection settings which were not designed for 512kb of ram. I tuned many of those and was still getting an abort but after some instrumentation it turns out that the runtime allocates a 64kb buffer for both stdin and stdout. After reducing those buffers considerably, the interpreter no longer aborted! It didn't, however, print anything out - which concerned me.

After a fair amount of debugging, I still have no idea where stdout goes. It's certainly not the same place as printf, which makes it to the monitor. Once I had that figured out, I realised I had a functioning interpreter!

State of play

There's a Dockerfile for the whole build process:

Installs the prerequisites, Xtensa gcc port, Espressif IDF
Rebuilds Newlib
Installs an OCaml via OPAM, then builds the OCaml ESP32 bytecode runtime
Finally builds a simple Hello World OCaml project and builds an image

You should be able to then flash the resulting image with make flash if you have a dev board connected and have passed the USB-serial device through to the container with --device=/dev/ttyUSB0 (on Linux).

Short term TODOs

There are a couple of TODOs that probably need to be cleaned up or fixed. As I mentioned earlier, we could expand some of the hasgot tests to include functionality Espressif's Newlib build doesn't have and this would simplify some of the configure changes. Figuring out how to redirect stdout and stderr to the monitor would also be incredibly useful.

Longer term plans

A native compiler backend for Xtensa would mean we could produce a more compact and hopefully more performant image which would be very useful in environments with tight power budgets. Speaking of low-power, some kind of DSL for programming the ultra-low power core on the board would also probably be very useful.

In terms of networking, the Espressif IDF ships with an lwip port for networking but there are sufficiently low level interfaces available for the Wi-Fi device that could work with Mirage's tcpip direct driver.

Any volunteers?

Sadiq Jaffer

Free Hardware Performance Counters in OxCaml Runtime Events

Why hardware performance counters?

The implementation

Runtime

OCaml API side

Taking it for a spin

GC phases by instructions-per-clock

Beyond the GC — profiling your own code

How expensive is it?

Trying it yourself

Next steps

Earth Observation on a Budget: Finding Solar Farms with a 42k-Parameter Model

Three Steps for OCaml to Crest the AI Humps

Can a model trained on satellite data really find brambles on the ground?

Adding an OCaml GC debugging task to terminal-bench

Last three months in OCaml (July 2025)

AI Coding Agents

Benchmarking

Agentic tooling

The OCaml runtime

Projects

Letting Claude Code check with Gemini

Examples

Qwen3 Leads the Pack: Evaluating how Local LLMs tackle First Year CS OCaml exercises

Terminology

Models

Results

High-level takeaways

What went wrong?

Syntax errors

Type system confusion

Hallucinated functions

Recursion

Limitations

Summary

A dataset of OCaml's Opam

JSON output from Deepseek R1 and distills with llama.cpp

Teaching Bloom Filters new tricks

Bloom Filters

Trade-offs

False positives

How they work

Adding to a filter

Querying a filter

Why they work

False positive probability for k=1

Generalising to multiple hash functions

Generalising Bloom filters

Fitting the traditional filter

Maxmin-based Bloom filters

Generalising this model further

A multi-member Bloom Filter

Recap and conclusion

Getting OCaml running on the ESP32

Caml runtime

Rebuilding Newlib

Building an image

Debugging

State of play

Short term TODOs

Longer term plans

End result