Richard Startin’s Blog

How Async-Profiler’s crash handler works

2023-11-12T00:00:00+00:00

Raw pointers in C/C++ open up entire classes of error that are practically unimaginable in higher level languages. So why does anybody use them at all? Unfortunately, it’s impossible to write a profiler without getting close to some of the sharp edges of unsafe memory. The async-profiler code base contains a lot of low level tricks, and it’s worth studying how some of them work.

This post is about how async-profiler safely dereferences arbitrary pointers without crashing the JVM, and why it needs to do that sometimes.

Signals and signal handlers
Frame pointer unwinding
SafeAccess::load
Replacing crash handlers

Signals and signal handlers

Linux allows threads to be interrupted by signals, which are identified by a signal number. Signals can be sent to threads via a number of syscalls, such as tgkill, or indirectly via profiling services like itimer or perf_event_open. When a thread transitions between the kernel and user space, such as when the thread is scheduled to run on the CPU, the kernel checks if there are any pending signals for the thread. If there is a pending signal, and the signal isn’t blocked by a signal mask, and there is a signal handler installed for the signal, then the thread is halted and the signal handler is invoked. A signal based sampling profiler consists of a mechanism for choosing threads and sending signals (usually SIGPROF) to them, then a signal handler for collecting information about the state of the thread (e.g. unwinding the stack). Signals and signal handlers are used for a number of other things, such as killing the process (SIGKILL), or signalling that something has gone wrong (SIGSEGV, SIGABRT, SIGBUS). If you just want your program not to crash when performing memory unsafe operations, you can install signal handlers for error signals and swallow them, though this will cause other problems without care.

Frame pointer unwinding

You don’t need to handle segfaults if your code can’t segfault, so why not just use a programming language with a compiler which verifies memory safety? Unfortunately, you wouldn’t have much of a profiler if you did that (without unsafe blocks, anyway). Consider the simplest stack unwinding approach using frame pointers.

Signal handlers are invoked with a ucontext which consists of a program counter, a frame pointer, and a stack pointer.

The program counter (pc) is the address of the code currently being executed. It can be used to find the currently executing function by binary searching a compiled binary’s text section (where the code is found) or the JVM’s code heap for JIT compiled code. It is enough to determine the currently executing instruction (which is how instruction profiling works). However, the program counter is not enough to determine the caller of the interrupted function.

The frame pointer (fp) determines the start address of the stack frame, which contains function parameters and local variables. The stack frame is like a snapshot of the registers before calling a function, so that when the called function returns, the calling context can be restored. Obtaining the frame pointer to the calling frame from the callee frame is known as unwinding. If a binary is compiled with frame pointers, it’s really simple: on x64, the rbp register contains the caller’s frame pointer. So frame pointer unwinding consists of the following (on x64, the registers differ on aarch64):

Record the program counter from the ucontext (this can be used later to determine things like function name, line number, and the instruction)
Read the caller’s frame pointer from rbp
Check termination (this could be if the frame pointer is zero, if the frame pointer is outside the bounds of the stack, or in async-profiler if a Java frame has been found).
Restore the caller frame from the caller’s frame pointer (dereference the frame pointer).
Record the program counter (see point 1)
Go to step 2

This is about as fast and simple as stack unwinding gets, but the problem is lots of binaries are compiled without frame pointers. If a binary is compiled without frame pointers, rbp contains arbitrary data and isn’t safe to dereference. Depending on the exact value stored in the register, dereferencing it may produce a bogus frame but will probably segfault (generate a SIGSEGV signal). I am unaware of any way to determine that an arbitrary binary was compiled with frame pointers or not, so the only way to determine whether rbp is a frame pointer or not is to dereference it and hopefuly get a frame. This puts frame pointer unwinding outside the realm of compile time memory safety checks: either mitigate at runtime or don’t do it.

`SafeAccess::load`

Async-profiler uses the following mechanism:

Always dereference untrusted data via a function called SafeAccess::load.
Install a SIGSEGV handler.
Check in the SIGSEGV handler if the SIGSEGV signal’s pc came from SafeAccess::load. If it did, patch things up as if the function returned zero, otherwise pass the signal on.

There are some details which are worth paying attention to in the function declaration (source):

   #ifdef __clang__
   #  define NOINLINE __attribute__((noinline))
   #else
   #  define NOINLINE __attribute__((noinline,noclone))
   #endif
   ...
   NOINLINE __attribute__((aligned(16)))
    static void* load(void** ptr) {
        return *ptr;
    }

Firstly, the function can’t be inlined because of the compiler attributes applied to it, because if it were inlined, it wouldn’t be possible to determine whether the SIGSEGV’s pc came from the function or not. Next, the function is aligned so that a pc can be determined to be within the function’s code’s address range easily.

In the signal handler installed when starting the profiler, the following code executes:

    uintptr_t length = SafeAccess::skipLoad(pc);
    if (length > 0) {
        // Skip the fault instruction, as if it successfully loaded NULL
        frame.pc() += length;
        frame.retval() = 0;
        return;
    }

This code figures out how far to skip ahead to simulate returning from the function, having set the return value to zero. The way the number of instructions to skip is determined is a little cryptic (only for x64 below):

    static uintptr_t skipLoad(uintptr_t pc) {
        if ((pc - (uintptr_t)load) < 16) {
            return *(u16*)pc == 0x8b48 ? 3 : 0;  // mov rax, [reg]
        }
        return 0;
    }

Checking if the pc is in the address range of the load function is straightforward enough, but what on earth is 0x8b48? Dereferencing a pc gives a sequence of bytes corresponding to the instructions. Casting to a u16* truncates to the most significant 2 bytes, which has the effect of ignoring the register the value would be loaded into. The 0x8b48 check verifies that the pc is pointing at a mov operation, otherwise something has gone wrong. Finally, 3 bytes need to be skipped over: 2 for the load, and one for the destination register.

Replacing crash handlers

The JVM has its own SIGSEGV handler already, in fact, it’s quite important because it’s part of the uncommon trap mechanism for deoptimisation of speculative optimisations. This means Async-profiler must delegate to the JVM’s SIGSEGV handler in order not to break the JVM. It does this by returning a function pointer to the old handler when installing its own handler, and then delegates to it if the pc didn’t come from SafeAccess::load. The JVM doesn’t exactly like signals it handles being overridden like this, given the central importance of SIGSEGV signals to the lifecycle of JIT compiled code, and running with -Xcheck:jni will spit out angry warning messages about it. These messages are annoying but much better than simply dropping samples when the JVM is not executing Java code, because Java code can call native code, and this is sometimes where performance problems lie.

Understanding Request Latency with Profiling

2023-09-05T00:00:00+00:00

It can be hard to figure out why response times are high in Java applications. In my experience, people either apply a process of elimination to a set of recent commits, or might sometimes use profiles of the system to explain changes in metrics. Making guesses about recent commits can be frustrating for a number of reasons, but mostly because even if you pinpoint the causal change, you still might not know why it was a bad change and are left in limbo. In theory, using a profiler makes root cause analysis a part of the triage process, so adopting continuous profiling should make this whole process easier, but using profilers can be frustrating because you’re using the wrong type of profile for analysis. Lots of profiling data focuses on CPU time, but the cause of your latency problem may be related to time spent off CPU instead. This post is about Datadog’s Java wallclock profiler, which I worked on last year, and explores how to improve request latency without making any code changes, or even seeing the code for that matter.

This is a personal blog and everything written below is my own personal opinion.

From metrics to profiles
async-profiler
Wallclock profiling
1. Sampling strategy
2. Tracer managed thread filter
Latency investigation
1. Increasing the heap size
2. Increasing the number of vCPUs
What about cost?

From metrics to profiles

Suppose you have metrics which show you that request latency is too high for a particular endpoint. Here is the endpoint level latency timeseries for a slow endpoint GET /allocation-stalls (the name may create a sense of foreboding) in a very contrived demo Java application.

The response time is about 3-4s, which we would like to improve somehow without undertaking an R&D project. Unless you have detailed tracing, you might start looking at profiles to do this. OpenJDK has a built-in profiler called Java Flight Recorder (JFR), which Datadog has used historically for collecting CPU profiles from Java services. Here’s what a flamegraph of JFR’s jdk.ExecutionSample events looks like after being processed by Datadog’s profiling backend:

To the far left there is some JVM activity derived from JFR’s GC events. Next to that there are some stack traces for Jetty’s routing (see JettyHandler.doHandle()). To the right are stack traces for handling the requests in another thread pool (see ThreadPoolExecutor$Worker.run()). There’s not much context beyond the frame names, and the most we have to go on is JavaServer.lambda$allocationStall$28() which we can probably guess is related to handling GET /allocation-stalls requests by its name. Java code can be very generic and it’s not always straightforward to map frames to context by frame names alone, though in this case it’s easy enough. Expanding it gets us to where I think most people turn away from profiling, assuming it was as easy to find the relevant frame as in this contrived example:

This points to random number generation being the bottleneck, which you aren’t likely to speed up. You may be able to reduce the number of random numbers generated, but in this contrived example, this would require renegotiating requirements. Part of the problem is that this flamegraph doesn’t show the complete picture. Without having the full picture, we can’t be sure we know what the bottleneck is, and might just be looking at the frame we happen to measure the most. This is where async-profiler comes in.

async-profiler

Some time ago, Datadog started experimenting with using async-profiler instead of JFR’s execution sampler in its continuous profiling product. There were two primary motivations for doing this:

It’s better
1. async-profiler has a much better CPU profiler than JFR, primarily because it can use CPU time (or other perf events) to schedule samples, whereas JFR uses wall time and a state filter.
2. async-profiler has several approaches to unwinding native parts of the stack, whereas JFR does not unwind stacks with native frames.
3. async-profiler reports failed samples so even if it’s not possible to get a full stack trace, it doesn’t distort the weights of other samples.
We could modify the source code in a fork
1. It’s easy to make changes to the source code and get those changes shipped in a timely manner.
2. JFR doesn’t allow any contextualisation of samples, which makes presenting the data it reports in useful and engaging ways challenging. It was quite easy to extend async-profiler to support arbitrary, multi-dimensional labeling, whereas the process to get this functionality into JFR would be long-winded.

So async-profiler’s excellent codebase gave us a starting point for implementing some of the features we had in mind.

Below is a flamegraph of the same application as above recorded with Datadog’s customisation of async-profiler. Notice the ZGC activity to the left and compare it to the GC activity we could infer from JFR above, there’s more detail and it’s more obvious that the service has a GC problem.

Having access to the source code made it possible to add new fields to event types, such as correlation identifiers. Decorating samples with span ids makes it possible to link each sample to the endpoint being served, rather than guess based on the frame name (which only gets harder outside of contrived examples). This makes it possible to filter the flamegraph by endpoint instead of guessing based on the frame names:

The green frames to the left, which are JNI calls into the lz4-java library, are missing from the JFR flamegraph because JFR won’t unwind stacks when the JVM is executing native code. In this particular case, this doesn’t actually distort the flamegraph much, because LZ4 compression is very fast and doesn’t need as much CPU time as the random number generation we already know about, but if it were the other way around the JFR flamegraph would be very misleading.

Even though the flamegraph now shows a complete picture of CPU time, it still points towards optimising random number generation, which we can’t do. The problem is now that CPU time isn’t always enough to understand latency.

Wallclock profiling

As used in Datadog’s profiler, async-profiler’s CPU profiler uses CPU time to schedule sampling, which means time spent off-CPU will not be sampled. In Java Thread terms, a CPU profiler can only sample threads in the RUNNABLE state, and ignores threads in the BLOCKED, TIMED_WAITING, and WAITING states, but a wallclock profiler can sample threads in any state. async-profiler also has a wallclock profiler, but we found we needed to tailor it to our requirements.

Sampling strategy

The first issue we encountered was with overhead (for our own particular requirements, don’t be discouraged from using async-profiler’s wallclock profiler in a different context). CPU profiling is always very lightweight because the overhead is proportional to the number of cores, whereas for wallclock profiling, overhead scales with the number of threads. async-profiler’s wallclock profiler samples sets of up to 8 threads round-robin, and to keep the sampling interval consistent as the number of threads changes, will adjust the sampling interval to as low 100 microseconds. Unless the user provides a thread filter expression to control how many threads to sample, the overhead of this approach can be quite high. Using a predefined thread filter assumes the user knows where the problems are ahead of time, and they might do when doing offline/non-continuous profiling. It can also produce quite large recordings: in 60s we might record as many as 4.8 million samples, which, at about 64 bytes per JFR event, corresponds to about 250MiB.

We wanted smaller recordings and needed to be very conservative about overhead, so we changed the way threads are sampled. We implemented reservoir sampling of threads so we could take a uniform sample of a fixed number of threads at a constant sampling interval. This gives a much more manageable upper bound on overhead and recording size: in 60s we can sample 16 threads at 100Hz and record at most 96,000 samples, which is just under 6MiB of JFR events. This approach reintroduces the problem async-profiler solves with its interval adjustment when the number of threads changes over time because this changes the probability of sampling a thread (the reservoir is fixed size), but this can be solved by recording the number of live threads each sample interval and upscaling later.

Tracer managed thread filter

High thread counts also can also reduce sample relevance, without a thread filter. For latency analysis it would be ideal to bias towards threads on the critical path. There are two kinds of noncritical threads for typical applications:

idle threads in over-provisioned thread pools, each waiting for items to appear in a queue.
background threads such as metrics reporters.

In the screenshot below, at least half of the samples are of the 200 workers (which is too many for this application) waiting for work to arrive on a LinkedBlockingQueue.

This isn’t very useful information: while fine-tuning the size of this thread pool would be generally beneficial, the threads aren’t blocking the threads processing requests.

What’s worse is when the samples are rendered as a timeline of a particular trace, the coverage is quite sparse (the grey bars are PARKED walltime samples, the blue bars are CPU samples).

Sampling idle or background activity reduces the coverage of traces.

Both of these problems can be solved by only sampling threads which have an active trace context set up by the tracer. Firstly, tracing is designed to record request latency, so if there is a trace context associated with a thread for a time period, there is a good chance the work is being done to complete a request. Thanks to context propagation, which propagates trace context into thread pools, any thread which ends up working to complete the request can be sampled. Secondly, biasing sampling towards traced activity helps focus the profiler on explaining trace latency, so we have the situation where the tracer can tell you how long a request took, and there’s a good chance that there will be wallclock samples to explain why it took so long. Whenever we have too much work to do, we can do a better job by prioritising some tasks over others, and it’s the same for sampling threads.

This results in a flamegraph focused on latency-critical work:

and when we look at the timeline of a particular trace, we have much better coverage (i.e. more grey bars for more threads)

This is implemented by repurposing async-profiler’s thread filter, so that it is only applied to the wallclock profiler, and is controlled by the tracer. The current thread is added to a bitset of thread ids whenever the current thread has a trace associated with it, and removed from the bitset whenever the thread no longer has a trace associated with it. If you have used a tracing API, you will be familiar with spans (logical operations within traces) and scopes (activations of spans on a thread). The tracer pushes a scope to a stack whenever a span is activated on a thread, so that any subsequent spans can refer to the activated span as a parent. Modifying the bitset each time a scope is pushed or popped from the scope stack is the most obvious implementation, but would lead to very high overhead for certain async workloads. Instead, the bitset is updated only when the scope stack becomes empty or non-empty, which corresponds to whether there is some trace associated with the thread or not, which almost always reduces the number of bitset updates and never increases it. Finally, the sampler thread implements reservoir sampling over the bitset of traced threads, rather than over all threads.

Latency investigation

The wallclock profiler reports stack traces for any blocking off-CPU activity, which should tell you where the blocking operation happened, and why the blocking operation occurred by looking at the ancestor frames. It should have been possible to figure out why the request latency is so high from the screenshots so far, but here’s the timeline for a particular request trace again:

The long grey bars, with durations of about a second, all contain the frame ZPageAllocator::alloc_page_stall(), which blocks on PosixSemaphore::wait(), with causal ancestor frame ByteBuffer.allocate(int). In the flamegraph scoped to the endpoint, we can see that about 50% of the samples under JavaServer.lambda$allocationStall$29() are allocation stalls, so the CPU profile missed roughly half the activity:

Allocation stalls happen with ZGC, when the application threads try to allocate more memory than is currently available. The application thread blocks until the requested allocation can be performed, which incurs latency. ZGC is effectively applying backpressure on the application threads, throttling concurrent access to memory as if it were a bounded queue, and it does this rather than OOM which is the only alternative. Allocation stalls are a symptom of an undersized heap, and there are two choices for mitigation: allocate a larger heap, or make the application allocate less.

In this contrived case, there are 30 tasks allocating a pair of 50 MiB ByteBuffers in a threadpool of 10 threads, which translates to 3GiB per request and 1000MiB allocated at any time during the request, with only 1GiB allocated to the heap. Having identified and understood allocation stalls, we can see the heap is too small for the workload. We can’t reduce allocation pressure by reducing the size of the buffers without renegotiating requirements, and we can’t reduce it by object pooling without blocking on the pool (we have less memory than we need) and our buffer pool might be less efficient than ZGC.

There is also a JFR event jdk.ZAllocationStall which could be used instead of a wallclock profiler to determine that the heap is too small, but it does not collect stacktraces.

Increasing the heap size

Solving this problem should be a simple case of increasing the heap size so more memory can be allocated without stalling. The service should then become CPU bound, and it was already established that random number generation can’t be accelerated, so we wouldn’t be able to remove this bottleneck without redefining the workload.

The heap size was increased to 6GiB. Looking at the wallclock profile, we can see a change in thread state:

Before, in the top image, about half the time was in the RUNNABLE state and half in the PARKED state, afterwards the split is 3:1. The allocation stalls have gone, and none of the PARKED time is under the JavaServer.lambda$allocationStall$29() frame: the allocation stalls have disappeared. Looking at a trace of a particular request, the grey bars which represented parking have gone, and we only see the blue CPU samples:

It’s clear the way the JVM executes the code has changed as a result of being given more resources, but it hasn’t actually reduced latency.

Increasing the number of vCPUs

The clue about what to fix next is in the timeline for the request: the CPU samples are very sparse in time and across the threads.

We can also see a strange discrepancy between the amount of wall time sampled in the RUNNABLE state and the amount of sampled CPU time:

If the workload is now CPU bound, as we expect it to be, and it’s actually executing, then the CPU time and the wall time should be the same, but there’s 18 seconds of RUNNABLE wall time and less than 5s of CPU time. Though we don’t have a direct way to show it, this indicates that the threads are competing to get scheduled on too few vCPUs, and if we could allocate more CPU to the container, the responses should become CPU bound. As can be seen in the timeline, the workload is parallelised over 10 threads, but only 2 vCPUs were allocated. Increasing the number of vCPUs to 10, finally, reduces latency dramatically:

Though they will never correspond exactly, the sampled RUNNABLE wall time and CPU times have converged:

and the timeline for a particular trace shows 10 threads with denser blue CPU samples:

What about cost?

This probably isn’t very satisfying: the cost of the running service has been increased dramatically. However, the aim was to reduce request latency for a contrived workload that isn’t amenable to optimisations. When code is at its efficient frontier (or effectively at its efficient frontier without, say, incurring huge engineering costs), there is a tradeoff between latency and cost. If you don’t have an SLA, you can just run this workload on 2vCPUs and a 1GiB heap for very little cost, and though ZGC will throttle allocations and it will be slow, things will basically work. If you do have an SLA, and optimisation is impractical or impossible, and you can’t negotiate requirements, you’ll need to incur costs and allocate more compute resources. It’s quite rare for a Java service to be at its efficient frontier, and there’s usually low hanging fruit surfaced by continuous profiling which can sometimes reduce latency and cost simultaneously.

Using JFR and JMC to root cause a performance bug in JMC caused by a typo in JFR

2022-12-18T00:00:00+00:00

When you open a profile in JMC, it normally takes a few seconds, but there are some profiles that JMC struggles to load. This happens when a profile contains events which violate an assumption made in JMC’s parser: events on the same thread are almost always disjoint in their durations. When JMC parses a JFR file, it splits the events of the same type emitted on the same thread into lanes so that events within a lane are disjoint. JMC can handle overlapping events, but assumes this essentially never happens, and exhibits quadratic scaling when events overlap.

The good news is that unless you hack it into the JFR file generation in your own customisation of async-profiler, it’s practically impossible to generate events that violate this assumption. However, if you run this code:

    public static void main(String... args) throws Exception {
        if (args.length < 1) {
            System.err.println("provide a directory to write the JFR file to");
            System.exit(1);
        }
        try(Recording recording = new Recording()){
            recording.setName("slow");
            recording.setDumpOnExit(true);
            recording.setDestination(getOrCreateDirectory(args[0]).resolve("slow.jfr"));
            recording.enable("jdk.FileWrite");
            recording.start();
            generateFileEvents();
            
        }
    }

    private static void generateFileEvents() throws IOException {
        Path directory = Files.createTempDirectory("files");
        for (int i = 0; i < 100 * 1000; i++) {
            try (RandomAccessFile file = new RandomAccessFile(directory.resolve(i + "").toFile(), "rw")) {
               file.write(("data" + i).getBytes(StandardCharsets.UTF_8));
            }
        }
    }

    private static Path getOrCreateDirectory(String dir) throws IOException {
        Path destination = Paths.get(dir);
        if (!Files.exists(destination)) {
            Files.createDirectory(destination);
        }
        return destination;
    }

And then open the .jfr file in JMC, you will see this for quite some time (well over 30s):

Fortunately JMC itself can be profiled as it is loading a profile, using JFR with the command:

jcmd <pid of JMC> JFR.start settings=profile filename=jmcrecording.jfr duration=60s

After 60s a profile will be dumped containing execution samples of whatever JMC spent its time doing. Loading the profile of JMC itself and navigating to Method Profiling:

This shows you a flamegraph of CPU samples:

There are over 6000 samples, which is a good number for a JFR CPU profile, which takes samples at about 100Hz. If most activity is during loading the profile is on the same thread, at 100Hz, having 6000 samples means we got one sample every 10ms so don’t have a lot of failed samples, which JFR doesn’t otherwise account for. Assuming there is no JNI activity, this means the profile should be reasonably accurate. It’s tempting to scroll to the bottom and look for the widest frame and optimise it. The obvious candidate here are the calls to ImpreciseScaleFactor.targetNumber, outlined in black below:

Though there is a little fat to trim in this method, this is a red herring; optimising this method would be a waste of time and it would take an heroic effort to move the needle by focusing here. CPU sampling can’t differentiate between a method being slow (requiring constant factors improvements) and being called too often (requiring algorithmic improvements). Unfortunately, JFR offers no way to perform a differential diagnosis to rule out constant factors and focus on algorithmic issues.

Once you have identified a bottleneck, the easiest way to differentiate between a constant factors and algorithmic issue is honestly just to read the code for a few of the frames towards the bottom of the stacktrace. Sometimes, this is too complicated, and I have resorted to inserting counter probes in these methods with bytebuddy (but I recently discovered async-profiler can do this and I have adopted this approach in preference to instrumentation during analyses). If you have a superlinear algorithm, you see high hit counts at the leaves, and low hit counts in the interior nodes. In any case, reading the code for DisjointBuilder.add (outlined in black below):

reveals what looks like an optimisation:

    // at 77580fe1b483b4daf8f46938c297185b01f32304
	public void add(T e) {
		IQuantity start = startAccessor.getMember(e);
		IQuantity end = endAccessor.getMember(e).in(start.getUnit());
		if (noLanes == 0) {
			addToNewLane(e, start, end);
		} else if (!lanes[0].accept(e, start, end)) {
			int changedLane = addToOtherLane(e, start, end);
			sortLanes(changedLane);
		}
	}

	private void sortLanes(int fromIndex) {
		// Sorting the lanes by descending end time
		for (int i = fromIndex; i > 0; i--) {
			if (lanes[i].end.compareTo(lanes[i - 1].end) > 0) {
				DisjointArray<T> tmp = lanes[i - 1];
				lanes[i - 1] = lanes[i];
				lanes[i] = tmp;
			}
		}
	}

	private int addToOtherLane(T e, IQuantity start, IQuantity end) {
		// Try with the other existing lanes
		for (int i = 1; i < noLanes; i++) {
			if (lanes[i].accept(e, start, end)) {
				return i;
			}
		}
		return addToNewLane(e, start, end);
	}

	private int addToNewLane(T e, IQuantity start, IQuantity end) {
		if (noLanes >= lanes.length) {
			lanes = Arrays.copyOf(lanes, (lanes.length * 3) / 2 + 2);
		}
		lanes[noLanes] = new DisjointArray<>(e, start, end);
		return noLanes++;
	}

lanes[0].accept is obviously expected to be the common case (it happens whenever the events are disjoint). Its implementation appears to be a constant time operation too, assuming the comparator isn’t bizarre:

    boolean accept(T e, IQuantity start, IQuantity end) {
        if (size >= array.length) {
            int newCapacity = array.length < 100 ? array.length * 4 : (array.length * 3) / 2 + 1;
            array = Arrays.copyOf(array, newCapacity);
        }
        if (start.compareTo(this.end) >= 0) {
            array[size++] = e;
            this.end = end;
            return true;
        }
        return false;
    }

The method addToOtherLane shows up in the flamegraph (outlined in black):

So we must be going down the other branch:

    int changedLane = addToOtherLane(e, start, end);
    sortLanes(changedLane);

addToOtherLane does a linear search for the first lane for which the last event ends before the event being added. This can obviously be improved by replacing the linear search with binary search, since the sort order of the lanes by end time is maintained. This doesn’t describe all of the samples though, and it seems strange that so many samples should be in MemberAccessorToolkit$2.getMember, and where is sortLanes?

This only makes sense if sortLanes (which is a quadratic bubble sort) has been inlined. Knowing that the lanes are maintained in sort order, when a lane needs to move, it’s possible to find where it needs to move to in logarithmic time, and then perform a linear time copy. This makes the algorithm linear in the number of lanes.

These changes were implemented in JMC-7950, which will be released in as part of JMC 9, and the issue is more or less fixed. If you open the profile generated by the code at the top of the post in JMC 9, it will still be slower to load than you might have hoped, but it will be several times faster. Using JFR to take a profile of JMC loading the same profile after the change produces a much smaller number of samples (JFR also records much less CPU time) and shows different stack traces:

Erik Gahlin pointed out on the issue that jdk.FileWrite events making profiles slow to load must be because of a bug in JFR. It turned out there has been a bug in JFR since JDK15, introduced here. The problematic change is here:

     @SuppressWarnings("deprecation")
     @JIInstrumentationMethod
     public void write(byte b[]) throws IOException {
-        FileWriteEvent event = FileWriteEvent.EVENT.get();
-        if (!event.isEnabled()) {
+        EventHandler handler = Handlers.FILE_WRITE;
+        if (!handler.isEnabled()) {
             write(b);
             return;
         }
+        long bytesWritten = 0;
+        long start = 0;
         try {
-            event.begin();
+            start = EventHandler.timestamp();
             write(b);
-            event.bytesWritten = b.length;
+            bytesWritten = b.length;
         } finally {
-            event.path = path;
-            event.commit();
-            event.reset();
+            long duration = EventHandler.timestamp(); // <-- bug here
+            if (handler.shouldCommit(duration)) {
+                handler.write(start, duration, path, bytesWritten);
+            }
         }
     }

The duration should have the start time subtracted from it. This means that these events will generally overlap because end = start + duration makes a nonsense value. The problem is compounded because the faulty duration will always exceed the latency threshold, which means these events flood JFR files to the extent they stress the parsing code.

Evaluating Equality Predicates

2022-12-18T00:00:00+00:00

I have just implemented support for (in)equality queries against a RangeBitmap, a succinct data structure in the RoaringBitmap library which supports range queries. RangeBitmap was designed to support range queries in Apache Pinot (more details here) but this enhancement would allow a range index to be used as a fallback for (in)equality queries in case nothing better is available. Supporting (in)equality queries allows a RangeBitmap to be used as a kind of compact inverted index, trading space for time, capable of supporting high cardinality gracefully.
Since RangeBitmap supports memory mapping from files, I think that it could be used for data engineering beyond Apache Pinot.

How to use it?
How does it work?

How to use it?

This post extends the example set up in Evaluating Range Predicates and Counting over Range Predicates, which centre around a large collection of Transaction objects:

  public static final class Transaction {
    private final int quantity;
    private final long price;
    private final long timestamp;

    public Transaction(int quantity, long price, long timestamp) {
      this.quantity = quantity;
      this.price = price;
      this.timestamp = timestamp;
    }

    public int getQuantity() {
      return quantity;
    }

    public long getPrice() {
      return price;
    }

    public long getTimestamp() {
      return timestamp;
    }
  }

Let’s find all the transactions with a certain quantity in the most obvious way possible, using the stream API.

    transactions.stream()
         .filter(transaction -> transaction.quantity == qty)
         .forEach(this::processTransaction);

On my laptop, it takes about 3ms to select about 100 transactions from 1M.

Benchmark	Mode	Threads	Samples	Score	Score Error (99.9%)	Unit	Param: minPrice	Param: minQuantity	Param: size
stream	avgt	1	5	2849.444527	30.581160	us/op	100	1	1000000

With a RangeBitmap, these same transactions can be found with the already existing between method:

    RoaringBitmap matchesQuantity = qtyIndex.between(qty - minQty, qty - minQty);
    matchesQuantity.forEach((IntConsumer) i -> processTransaction(transactions.get(i)));

as explained in Evaluating Range Predicates, the quantities in the index have been anchored to the smallest value in the population, as a size optimisation.

This performs quite a lot better, selecting the same transactions in under 300us, which is a 10x improvement.

Benchmark	Mode	Threads	Samples	Score	Score Error (99.9%)	Unit	Param: minPrice	Param: minQuantity	Param: size
stream	avgt	1	5	2849.444527	30.581160	us/op	100	1	1000000
between	avgt	1	5	296.488277	0.703309	us/op	100	1	1000000

This can be rewritten using eq:

    RoaringBitmap matchesQuantity = qtyIndex.eq(qty - minQty);
    matchesQuantity.forEach((IntConsumer) i -> processTransaction(transactions.get(i)));

The new eq method doesn’t need to do as much work as between, which needs to maintain and combine two bitsets during the scan over the RangeBitmap, whereas eq only needs one. This means we can get a good speedup from eq to select the same transactions:

Benchmark	Mode	Threads	Samples	Score	Score Error (99.9%)	Unit	Param: minPrice	Param: minQuantity	Param: size
stream	avgt	1	5	2849.444527	30.581160	us/op	100	1	1000000
between	avgt	1	5	296.488277	0.703309	us/op	100	1	1000000
eq	avgt	1	5	183.913329	1.994362	us/op	100	1	1000000

It’s worth making a comparison with an inverted index over quantity (so a bitmap of transaction positions per quantity) now. The inverted index would always win for speed on this query, but can take up a lot more space, and the inverted index would almost always lose for range queries. A range encoded inverted index (a mapping from quantity to bitmap of the positions of all transactions with a smaller quantity) will generally beat RangeBitmap for speed at the cost of space. This makes a RangeBitmap suitable for equality queries eiter on a high cardinality attribute or as a fallback better than scanning when range queries are more common.

Inequality filters tend not to be very selective, so benchmarking the evaluation would be dominated by the time to scan the results, but the neq method is present for API symmetry.

In the same set of changes there are also methods to push a RoaringBitmap context down into eq and neq queries, which behaves like an intersection. Rather than producing a large bitmap and then intersecting it with a small context bitmap, the context bitmap is used to potentially skip over large sections of the RangeBitmap. There are also eqCardinality and neqCardinality methods, which produce counts rather than bitmaps, as described for range counts.

How does it work?

RangeBitmap has a simple two-dimensional structure. The first dimension is the rows in the order the values were added in. The second dimension is the binary representation of each value added, after range encoding, and there are only as many of these columns as there are significant bits in the largest value added.

The layout is essentially the bit-transposition of the values added, striped in bands of $2^{16}$ rows, with a little bit of metadata to help traversals. Each $2^{16}$ rows are bucketed into a horizontal slice which consists of $64 - clz(max)$ RoaringBitmap containers, one for each bit of the inputs, and a mask of size $64 - clz(max)$ bits, indicating the presence of a container. If the $n$th value added to the RangeBitmap does not have bit $i$ set, the $i$th container in the $n/2^{16}$th slice will have no bit set. If all $2^{16}$ values in the slice have bit $i$ unset, the mask will have bit $i$ unset and no container, which is an optimisation to avoid storing empty containers.

If the following values (with their binary representation in brackets) are added in sequence

(101010)
(011000)
(001001)
(011011)

The values are first negated to range encode them (see here for explanation).

There will be one horizontal slice with four rows in it, a 6 bit mask, and 5 containers per slice because the 4th bit is present in all the values in the slice. This looks something like this, if stored as plain bitsets:

110111 1100 0110 1111 1010 0110

There are three kinds of RoaringBitmap containers for different densities - sparse, dense, and run length encoded. Containers with only four values would always be sparse, for the sake of explanation, assume these are just the first four values in a much larger slice. Encoded as containers, this looks like:

110111 [4,2] {2,4} [0,4] 1010 {2,4}

Where the containers in braces have been represented as arrays of 16 bit values, the ones in square brackets are run length encoded, and the binary numbers are just bitsets.

To evaluate queries against RangeBitmap, the slices need to be iterated over in ascending order. For each slice, the containers need to be extracted and combined, using a different combination algorithm for each relation. The equality combination algorithm is very simple:

start with a bitset b with a set bit for each row in the slice
for each i from 0 to the size of the slice’s mask,
- if i is set in the query value, remove the container i’s bits (or none, if container i is missing for the current slice) from b
- If i is absent in the query value, intersect container i’s bits with b (or just clear it id the container is missing for the slice)

For inequality, b needs to be complemented at the end. Range queries are generally more complex to evaluate than this, but all queries are evaluated in this slice by slice in ascending row order fashion.

Counting over Range Predicates

2022-03-27T00:00:00+00:00

This post follows on from my last post about selecting objects satisfying a range predicate, and instead looks at how to count the objects. If you can select objects, you can count them too, but it’s a simpler problem so resources can be saved with a specialised solution.

Suppose you want to count how many of the Transaction objects below satisfy complex filters, and don’t care about details about the objects themselves.

  public static final class Transaction {
    private final int quantity;
    private final long price;
    private final long timestamp;

    public Transaction(int quantity, long price, long timestamp) {
      this.quantity = quantity;
      this.price = price;
      this.timestamp = timestamp;
    }

    public int getQuantity() {
      return quantity;
    }

    public long getPrice() {
      return price;
    }

    public long getTimestamp() {
      return timestamp;
    }
  }

Once again, this sounds like a task for a database, but imagine you don’t have one to rely on.

You might use the streams API:

    transactions.stream()
    .filter(transaction -> transaction.quantity >= qty && transaction.price <= price
                    && transaction.timestamp >= begin && transaction.timestamp <= end)
            .count();

This will inspect every transaction to check if it matches the constraints, and will count the number of matches. For 1M transactions, with parameters which select about 250 transactions, this takes about 10ms on my laptop:

Benchmark	Mode	Threads	Samples	Score	Score Error (99.9%)	Unit	Param: minPrice	Param: minQuantity	Param: size
stream1	avgt	1	5	9721.806835	27.935884	us/op	100	1	1000000

It took less time to actually select the objects, which, without any knowledge of the implementation, suggests that the objects are probably selected in order to be counted. The transactions are the same as those in the previous post, and sorted by time. This means reordering the conditions to the time filter is applied first is as effective as it was last time by reducing the number of branch misses.

    transactions.stream()
    .filter(transaction -> transaction.timestamp >= begin && transaction.timestamp <= end
                    && transaction.quantity >= qty && transaction.price <= price)
            .count();

Benchmark	Mode	Threads	Samples	Score	Score Error (99.9%)	Unit	Param: minPrice	Param: minQuantity	Param: size
stream1	avgt	1	5	9721.806835	27.935884	us/op	100	1	1000000
stream2	avgt	1	5	3283.291345	44.566458	us/op	100	1	1000000

Something easy to ignore in a Stream API benchmark is how much better the performance is when there is only one implementation of the filter, which allows the filter to be inlined. This makes the code in the benchmark artificially simple, when really the whole point of the API is that many filters would be used, which is a triumph of modularity over efficiency (but you should use the right tool for the job and know when such a difference matters). Artificially polluting the filter in the benchmark setup shows how artificial the result for stream2 was:

    @Setup(Level.Trial)
    public void setup() {
      super.setup();
      transactions.stream()
          .filter(transaction -> transaction.quantity < 0 && transaction.price > 1
              && transaction.timestamp >= 1 && transaction.timestamp <= Long.MAX_VALUE - 1)
          .count();
      transactions.stream()
          .filter(transaction -> transaction.quantity < 10 && transaction.price > 1000
              && transaction.timestamp >= -1 && transaction.timestamp <= Long.MAX_VALUE - 1)
          .count();
    }

Benchmark	Mode	Threads	Samples	Score	Score Error (99.9%)	Unit	Param: minPrice	Param: minQuantity	Param: size
stream1	avgt	1	5	9721.806835	27.935884	us/op	100	1	1000000
stream2	avgt	1	5	3283.291345	44.566458	us/op	100	1	1000000
stream2Polluted	avgt	1	5	7801.823477	41.229256	us/op	100	1	1000000

Inlining or no inlining, it’s inefficient to use linear search if the transactions are sorted, which they are (by time). As far as I’m aware there’s no way to communicate to the Stream API that a collection is already sorted and can be skipped over, and it doesn’t know how to crack filter predicates to exploit this anyway. So avoiding the Stream API and writing old fashioned Java code is effective. In the benchmark data, the time range predicate selects about 10% of the data, so the other two predicates should only apply to this much data. Assume the timestamps are unique for simplicity’s sake:

    int first = Collections.binarySearch(transactions, new Transaction(0, 0, begin),
            Comparator.comparingLong(Transaction::getTimestamp));
    first = first < 0 ? -first - 1 : first; 
    int last = Collections.binarySearch(transactions, new Transaction(0, 0, end),
            Comparator.comparingLong(Transaction::getTimestamp));
    last = last < 0 ? -last - 1 : last;
    long count = 0;
    for (int i = first; i <= last; i++) {
      Transaction transaction = transactions.get(i);
      if (transaction.quantity >= qty && transaction.price <= price) {
        count++;
      }
    }

Benchmark	Mode	Threads	Samples	Score	Score Error (99.9%)	Unit	Param: minPrice	Param: minQuantity	Param: size
stream1	avgt	1	5	9721.806835	27.935884	us/op	100	1	1000000
stream2	avgt	1	5	3283.291345	44.566458	us/op	100	1	1000000
stream2Polluted	avgt	1	5	7801.823477	41.229256	us/op	100	1	1000000
binarySearch	avgt	1	5	786.482285	1.994657	us/op	100	1	1000000

Sorting is not included in the benchmark time, but data is often naturally sorted by time. It’s not a lot more code but goes a lot faster, and easier to measure because it’s so simple. JMH’s perfnorm profiler shows that there is still a good number of branches missed, despite shedding about 90% of the work. This means the instructions per cycle (IPC) is low:

Benchmark	Mode	Threads	Samples	Score	Score Error (99.9%)	Unit	Param: minPrice	Param: minQuantity	Param: size
binarySearch	avgt	1	5	786.482285	1.994657	us/op	100	1	1000000
binarySearch:IPC	avgt	1	1	0.755879	NaN	insns/clk	100	1	1000000
binarySearch:branch-misses	avgt	1	1	50739.339319	NaN	#/op	100	1	1000000
binarySearch:branches	avgt	1	1	434330.541115	NaN	#/op	100	1	1000000

This can be fixed by complicating the filter in the scan somewhat.

    long count = 0;
    for (int i = first; i <= last; i++) {
      Transaction transaction = transactions.get(i);
      count += (Math.min(1, Math.max(transaction.quantity - qty, 0)) 
        + Math.min(1, Math.max(price - transaction.price, 0))) >>> 1;
    }

If the transaction’s quantity is greater than the threshold, Math.max(transaction.quantity - threshold, 0) will be positive, wrapping this in Math.min(1, x) means that the count will be one if the predicate holds and zero if it doesn’t. This can be added to the other condition and divided by two so the count will be incremented by one when both conditions hold, and by zero when they don’t.

Math.max and Math.min are special and get compiled to conditional move instructions which aren’t speculated so can’t miss. They are annotated as @IntrinsicCandidate:

    @IntrinsicCandidate
    public static int max(int a, int b) {
        return (a >= b) ? a : b;
    }

    @IntrinsicCandidate
    public static int min(int a, int b) {
        return (a <= b) ? a : b;
    }

This means if you have a hot loop, they should be preferred to inlining the equivalent logic, which may or may not be compiled to the same code. They’re also more readable, in my opinion. The branch misses basically disappear, and the instructions per cycle is now much higher:

Benchmark	Mode	Threads	Samples	Score	Score Error (99.9%)	Unit	Param: minPrice	Param: minQuantity	Param: size
binarySearch	avgt	1	5	786.482285	1.994657	us/op	100	1	1000000
binarySearch:IPC	avgt	1	1	0.755879	NaN	insns/clk	100	1	1000000
binarySearch:branch-misses	avgt	1	1	50739.339319	NaN	#/op	100	1	1000000
binarySearch:branches	avgt	1	1	434330.541115	NaN	#/op	100	1	1000000
binarySearchBranchFreeScan	avgt	1	5	299.725900	2.414697	us/op	100	1	1000000
binarySearchBranchFreeScan:IPC	avgt	1	1	3.278388	NaN	insns/clk	100	1	1000000
binarySearchBranchFreeScan:branch-misses	avgt	1	1	524.965594	NaN	#/op	100	1	1000000
binarySearchBranchFreeScan:branches	avgt	1	1	403042.075820	NaN	#/op	100	1	1000000

However, I am fairly certain that no analytical database written in Java implements counting over predicates like this. This is because fusing the predicate with aggregation is hostile to modularity, though generating code specialised to the problem is a good option if the implementation needs to be both fast and modular. It also won’t work with arbitrary numbers of predicates and needs to be padded to the next power of 2 so a matching result can be shifted down to one or zero in order to avoid integer division. If the JIT compiler were smarter and could inline the predicate evaluation into the count operator, I imagine it could transform modular code into this form, but it would probably get it wrong as often as it gets it right. So this is probably an artificially good result, but it is the best so far.

Benchmark	Mode	Threads	Samples	Score	Score Error (99.9%)	Unit	Param: minPrice	Param: minQuantity	Param: size
stream1	avgt	1	5	9721.806835	27.935884	us/op	100	1	1000000
stream2	avgt	1	5	3283.291345	44.566458	us/op	100	1	1000000
stream2Polluted	avgt	1	5	7801.823477	41.229256	us/op	100	1	1000000
binarySearch	avgt	1	5	786.482285	1.994657	us/op	100	1	1000000
binarySearchBranchFreeScan	avgt	1	5	299.725900	2.414697	us/op	100	1	1000000

In my last post I used RangeBitmap - which is the data structure used in Apache Pinot’s range index - to select transactions matching the predicate. Combined with binary search to apply the time range filter, it was much faster than any other implementation I measured.

I have recently implemented the ability to count values matching a predicate, instead of producing a bitmap of their offsets. The API is symmetric with the selection API:

  long index(int qty, long price, long begin, long end) {
    RoaringBitmap inTimeRange = timestampIndex.between(begin, end);
    RoaringBitmap matchesQuantity = quantityIndex.gte(qty, inTimeRange);
    return priceIndex.lteCardinality(price, matchesQuantity);
  }
  
  long binarySearchThenIndex(int qty, long price, long begin, long end) {
    int first = Collections.binarySearch(transactions, new Transaction(0, 0, begin),
            Comparator.comparingLong(Transaction::getTimestamp));
    int last = Collections.binarySearch(transactions, new Transaction(0, 0, end),
            Comparator.comparingLong(Transaction::getTimestamp));
    RoaringBitmap inTimeRange = RoaringBitmap.bitmapOfRange(first, last + 1);
    RoaringBitmap matchesQuantity = quantityIndex.gte(qty, inTimeRange);
    return priceIndex.lteCardinality(price, matchesQuantity);
  }

RangeBitmap is likely to be suboptimal for counting because the data structure is optimised for producing a sorted output set, which isn’t required for counting, but if you have a data stucture to support selection it ought to be able to provide a count without materialising intermediate results. For selection, just using an index for all three attributes and combining the results was a lot faster than binarySearch but for counting it’s a little disappointing at the moment. I haven’t tried to optimise the counting methods yet so there is probably a lot of low-hanging fruit. Cutting the range down with the binary search first produces a good result, even better than the hard to modularise binarySearchBranchFreeScan:

Benchmark	Mode	Threads	Samples	Score	Score Error (99.9%)	Unit	Param: minPrice	Param: minQuantity	Param: size
stream1	avgt	1	5	9721.806835	27.935884	us/op	100	1	1000000
stream2	avgt	1	5	3283.291345	44.566458	us/op	100	1	1000000
stream2Polluted	avgt	1	5	7801.823477	41.229256	us/op	100	1	1000000
binarySearch	avgt	1	5	786.482285	1.994657	us/op	100	1	1000000
binarySearchBranchFreeScan	avgt	1	5	299.725900	2.414697	us/op	100	1	1000000
index	avgt	1	5	666.951907	14.861610	us/op	100	1	1000000
binarySearchThenIndex	avgt	1	5	78.191544	17.903472	us/op	100	1	1000000

RangeBitmap was designed for use in Apache Pinot, so is compressed and supports zero-copy mapping to and from disk. There are more details about how it works in depth in RangeBitmap - How range indexes work in Apache Pinot.

The benchmark used in this post is not particularly scientific but is here. If you want to run it, you will get different numbers, but the rank of each benchmark score should not change.

Evaluating Range Predicates

2022-03-12T00:00:00+00:00

Suppose you are doing some kind of data analysis in Java, perhaps you are analysing transactions (as in sales made). You have complex filters to evaluate before performing a calculation on Transaction objects.

  public static final class Transaction {
    private final int quantity;
    private final long price;
    private final long timestamp;

    public Transaction(int quantity, long price, long timestamp) {
      this.quantity = quantity;
      this.price = price;
      this.timestamp = timestamp;
    }

    public int getQuantity() {
      return quantity;
    }

    public long getPrice() {
      return price;
    }

    public long getTimestamp() {
      return timestamp;
    }
  }

Suppose you want to find all transactions in a time range, and where the quantity exceeds a threshold but the price is lower than another threshold. This is clearly work that could be pushed down into a database, but you may not be able to do this for a couple of possible reasons:

You don’t have huge data volumes but your management may have been convinced not to procure a proper database with a SQL interface
You may not have a database because you have a lot of data, and it’s cheaper to store it in something like S3, you have processing jobs which filter the data programmatically

For whatever reason, you don’t have a database with a SQL interface and have to do the filtering yourself in a Java program, how might you do it?

You might use the streams API:

    transactions.stream()
    .filter(transaction -> transaction.quantity >= qty && transaction.price <= price
                    && transaction.timestamp >= begin && transaction.timestamp <= end)
            .forEach(this::processTransaction);

This will inspect every transaction to check if it matches the constraints. For 1M transactions, with parameters which select about 250 transactions, this takes about 10ms on my laptop:

Benchmark	Mode	Threads	Samples	Score	Score Error (99.9%)	Unit	Param: minPrice	Param: minQuantity	Param: size
stream1	avgt	1	5	8885.846120	41.192708	us/op	100	1	1000000

Suppose the transactions are actually sorted by time, the timestamp conditions should be predictable but aren’t evaluated first. This means the unpredictable price and quantity conditions are evaluated for every transaction. Reordering the conditions halves the runtime!

Benchmark	Mode	Threads	Samples	Score	Score Error (99.9%)	Unit	Param: minPrice	Param: minQuantity	Param: size
stream1	avgt	1	5	8885.846120	41.192708	us/op	100	1	1000000
stream2	avgt	1	5	3859.494033	42.012070	us/op	100	1	1000000

This is an interesting result, and JMH’s perfnorm profiler explains why:

Benchmark	Mode	Threads	Samples	Score	Score Error (99.9%)	Unit	Param: minPrice	Param: minQuantity	Param: size
stream1	avgt	1	5	8885.846120	41.192708	us/op	100	1	1000000
stream1:branches	avgt	1	5	4537456.732			100	1	1000000
stream1:branch-misses	avgt	1	5	505305.308			100	1	1000000
stream1	avgt	1	5	3859.494033	42.012070	us/op	100	1	1000000
stream2:branches	avgt	1	5	4648262.253			100	1	1000000
stream2:branch-misses	avgt	1	5	51040.523			100	1	1000000

There are about the same number of branches in each case, but ~10x fewer branch misses. Branch misses are expensive!

However, if the transactions are sorted by time, linear search will be wasteful unless the timestamp range covers most of data. In my benchmark, it covers about 10% of the data, so binary searching for the first and last timestamps in the range could correspond to up to ~10x improvement. Assume the timestamps are unique for simplicity’s sake:

    int first = Collections.binarySearch(state.transactions, new Transaction(0, 0, begin),
            Comparator.comparingLong(Transaction::getTimestamp));
    first = first < 0 ? -first - 1 : first; 
    int last = Collections.binarySearch(state.transactions, new Transaction(0, 0, end),
            Comparator.comparingLong(Transaction::getTimestamp));
    last = last < 0 ? -last - 1 : last;
    for (int i = first; i <= last; i++) {
      Transaction transaction = state.transactions.get(i);
      if (transaction.quantity >= qty && transaction.price <= price) {
        bh.consume(transaction);
      }
    }

Benchmark	Mode	Threads	Samples	Score	Score Error (99.9%)	Unit	Param: minPrice	Param: minQuantity	Param: size
stream1	avgt	1	5	8885.846120	41.192708	us/op	100	1	1000000
stream2	avgt	1	5	3859.494033	42.012070	us/op	100	1	1000000
binarySearch	avgt	1	5	1204.366198	8.455373	us/op	100	1	1000000

Sorting to ensure this is a valid application of binary search is not included in the benchmark time.

If several filters need to be evaluated against the collection, it would make sense to perform the sort to benefit from this.

This particular problem - applying range predicates to unsorted data - is solved by the RangeBitmap data structure in the RoaringBitmap library. The data structure is immutable and benefits from knowledge of the range of values in the data set, but if several filters need to be evaluated, building an index on each attribute could be worth it. The data structure has a build-then-use lifecycle:

    long minTimestamp = Long.MAX_VALUE;
    long maxTimestamp = Long.MIN_VALUE;
    long minPrice = Long.MAX_VALUE;
    long maxPrice = Long.MIN_VALUE;
    int minQty = Long.MAX_VALUE;
    int maxQty = Long.MIN_VALUE;
    for (Transaction transaction : transactions) {
        minTimestamp = Math.min(minTimestamp, transaction.getTimestamp());
        maxTimestamp = Math.max(maxTimestamp, transaction.getTimestamp());
        minPrice = Math.min(minPrice, transaction.getPrice());
        maxPrice = Math.max(maxPrice, transaction.getPrice());
        minQty = Math.min(minQty, transaction.getQuantity());
        maxQty = Math.max(maxQty, transaction.getQuantity());
    }
    var timestampAppender = RangeBitmap.appender(maxTimestamp - minTimestamp);
    var priceAppender = RangeBitmap.appender(maxPrice - minPrice);
    var qtyAppender = RangeBitmap.appender(maxQty - minQty);
    for (Transaction transaction : transactions) {
        timestampAppender.add(transaction.getTimestamp() - minTimestamp);
        priceAppender.add(transaction.getPrice() - minPrice);
        qtyAppender.add(transaction.getQuantity() - minQty);
    }
    var timestampIndex = timestampAppender.build();
    var priceIndex = priceAppender.build();
    var qtyIndex = qtyAppender.build();

Whether the two passes over the data or the half page of code are worth it depends on how many filters you need to do and how fast they need to be.

RangeBitmap produces a RoaringBitmap of the indexes which satisfy a predicate, and can take RoaringBitmap parameters as inputs to skip over rows already filtered out. The Streams API code used before is translated into RangeBitmap API calls:

    RoaringBitmap inTimeRange = timestampIndex.between(minTimeThreshold - minTime, maxTimeThreshold - minTime);
    RoaringBitmap matchesQuantity = qtyIndex.gte(minQtyThreshold - minQty, inTimeRange);
    RoaringBitmap matchesPrice = priceIndex.lte(maxPriceThreshold - minPrice, matchesQuantity);
    matchesPrice.forEach((IntConsumer) i -> processTransaction(transactions.get(i)));
  }

The anchoring by the minimum values for each attribute is a little convoluted but improves efficiency (unless the minimum value is zero anyway) and this would be better abstracted by a convenience class in a real application. For the same data this is ~2x faster than the binary search approach:

Benchmark	Mode	Threads	Samples	Score	Score Error (99.9%)	Unit	Param: minPrice	Param: minQuantity	Param: size
stream1	avgt	1	5	8885.846120	41.192708	us/op	100	1	1000000
stream2	avgt	1	5	3859.494033	42.012070	us/op	100	1	1000000
binarySearch	avgt	1	5	1204.366198	8.455373	us/op	100	1	1000000
index	avgt	1	5	690.562119	25.725005	us/op	100	1	1000000

Time spent building the index is not included in the benchmark time since it is assumed the filter will be applied several times and building the index will be amortised.

Binary search is much faster than the algorithm RangeBitmap uses, and it’s a shame it can only be used on one attribute since it’s not possible to do a global sort on more than one attribute. If the data is sorted by timestamp, a hybrid approach can be taken:

    int first = Collections.binarySearch(state.transactions, new Transaction(0, 0, begin),
            Comparator.comparingLong(Transaction::getTimestamp));
    first = first < 0 ? -first - 1 : first; 
    int last = Collections.binarySearch(state.transactions, new Transaction(0, 0, end),
            Comparator.comparingLong(Transaction::getTimestamp));
    last = last < 0 ? -last - 1 : last;
    RoaringBitmap inTimeRange = RoaringBitmap.bitmapOfRange(first, last + 1);
    RoaringBitmap matchesQuantity = qtyIndex.gte(maxPriceThreshold - minPrice, inTimeRange);
    RoaringBitmap matchesPrice = priceIndex.lte(minQtyThreshold - minQty, matchesQuantity);
    matchesPrice.forEach((IntConsumer) i -> processTransaction(state.transactions.get(i)));

This is over 100x faster than the original Streams API code and takes under 100us, all else being equal.

Benchmark	Mode	Threads	Samples	Score	Score Error (99.9%)	Unit	Param: minPrice	Param: minQuantity	Param: size
stream1	avgt	1	5	8885.846120	41.192708	us/op	100	1	1000000
stream2	avgt	1	5	3859.494033	42.012070	us/op	100	1	1000000
binarySearch	avgt	1	5	1204.366198	8.455373	us/op	100	1	1000000
index	avgt	1	5	690.562119	25.725005	us/op	100	1	1000000
binarySearchThenIndex	avgt	1	5	82.893010	12.486953	us/op	100	1	1000000

RangeBitmap was designed to power the range indexes in Apache Pinot, so is compressed and supports zero-copy mapping to and from disk. There are more details about how it works in depth in RangeBitmap - How range indexes work in Apache Pinot.

It’s likely you’re sensible and pushing your filters down to your database to evaluate, but this data structure may be useful in some applications.

The benchmark used in this post is not particularly scientific but is here. If you want to run it, you will get different numbers, but the rank of each benchmark score should not change.

RangeBitmap - How range indexes work in Apache Pinot

2022-03-07T00:00:00+00:00

Suppose you have an unsorted array of numeric values and need to find the set of indexes of all the values which are within a range. The range predicate will be evaluated many times, so any time spent preprocessing will be amortised, and non-zero spatial overhead is expected. If the data were sorted, this would be very easy, but the indexes of the values have meaning so the data cannot be sorted. To complicate the problem slightly, the set of indexes must be produced in sorted order.

These are the requirements for a range index in a column store, where the row indexes implicitly link values stored separately in different columns. The purpose of a range index is to accelerate a query like the one below which identifies all spans in a time window where the duration exceeds 10ms (in nanoseconds):

select 
  traceId, 
  spanId, 
  operationName 
from 
  spans 
where 
  duration > 10000000 
  and timestamp between '2022-03-06 02:00:00.000' 
  and '2022-03-06 03:00:00.000'

How can the spans be indexed to make this query as fast as it can be? Different databases solve this problem in different ways, but the context of this post is a column store like Apache Pinot.

It’s unusual for there to be only one important class of query, so other constraints need to be taken into account. The query above for slow spans isn’t the only query that needs to be fast, queries like the one below to compute the time spent in each operation of a trace also need to execute quickly:

select  
  serviceName,
  operationName,
  count(*),
  sum(duration)
from 
  spans 
where 
  traceId = 1234578910
group by 
  serviceName,
  operationName

Apache Pinot tables consist of segments which usually correspond to the data ingested from a stream within a time window, and will generally have the same implicit partitioning as the stream. Segments have a columnar layout, and contain their indexes, metadata, and other data structures. Segments are independent of each other and all data structures are scoped to their segment; data is sorted within the scope of a segment and not globally, and so on.

A Pinot cluster consists of four different classes of service which have different responsibilities, of which two are relevant to how to optimise this query. Servers are responsible for querying segments, they know how to read the segment format, load indexes, prune (avoid querying) and query the segments and merge segment-level query results into server-level query results. Brokers are responsible for routing queries to servers and merging server-level results, they can perform some amount of segment pruning to protect the servers’ resources.

The second query needing to be fast imposes constraints on what can be done to optimise the first query. Firstly, brokers can use knowledge of segment partitioning to optimise query routing. If we need the second query to be fast, we should partition on traceId which limits the number of servers the broker need to route the query to. Unless the trace’s duration exceeds the time taken to produce one segment (which may happen when tracing batch systems), all the spans in a trace should fit in a single segment, so partitioning by traceId should mean only a single segment need be queried. This means that the first query (find all the slow spans in the last hour) will not be able to use partitioning to prune segments and the query will be routed to all servers.

The second query will also need an index on traceId to be efficient because a typical segment will contain millions of rows, and scanning all these rows will be too slow. Choosing an appropriate index requires some knowledge of the data (though this decision can be automated given representative sample data). In case you are unfamiliar with tracing, traceId and spanId are high cardinality attributes and there is a hierarchical relationship between spans and traces: a trace is a collection of spans. We should expect that traces last at most minutes and consist of fewer than 1000 spans in the typical case.

Pinot has several types of index, but the choice here would be between a sorted index and an inverted index. Sorted indexes offer the best space/time tradeoff of Pinot’s indexes but, unsurprisingly, a sorted index can only be applied to a sorted column, and only one column can be sorted. This makes the choice of sorted column one of the biggest decisions when designing a Pinot table.

Given the hierarchical relationship between traceId and spanId, and an expectation that most traces will contain far fewer than 1000 spans, sorting on traceId indexes spanId for free because a scan for spanId would inspect only spans within the same trace. Creating an inverted index on traceId is a possibility, but it would have quite high cardinality: if there are 10M spans in a segment, and 100 spans per trace, then there would be 100k postings lists. Moreover, the performance of the aggregation in the second query depends on locality, which sorting improves. In this case, the case for sorting by traceId probably trumps sorting by duration or timestamp.

Fortunately, the timestamp filter can be used to prune segments if it is configured as the time column. Segments have metadata including the minimum and maximum value of the time column, and these time boundary values can be used to eliminate segments before querying them. The filter timestamp between '2022-03-06 02:00:00.000' and '2022-03-06 03:00:00.000' narrows the segments to query down to those with boundary values which intersect with the hour specified by the query. Assuming that each segment corresponds to 3 hours of ingested events within its partition, pruning by time boundary cuts the number of segments which need to be pruned from all segments ever to at most twice the number of partitions. However, within each of the unpruned segments, there are still millions of records which need to be filtered to find the slow spans in the time range, and the rows within the segments aren’t sorted by timestamp.

All of this is enough to justify the existence of a range index in a system like Pinot. This post is about the design and implementation of RangeBitmap, a data structure which powers Pinot’s new range indexes.

Implementing a Range Index
Bit slicing
RangeBitmap design
Usage in Apache Pinot
Links:

Implementing a Range Index

Zooming into the data now, imagine you have an array of durations like those below, which are all small numbers for the sake of simplicity:

$[10, 3, 15, 0, 0, 1, 5, 6, 2, 1, 12, 14, 3, 9, 11]$

We want to get the row indexes of the values which satisfy predicates so attributes at the same indexes can be selected. We also want the sets to be sorted so scans over other columns are sequential which is more efficient than traversing in random order. From the list above we would want the following outputs for each of the predicates below:

predicate	expected output
$x < 3$	$ \{3, 4, 5, 8, 9 \} $
$x < 10$	$ \{ 1, 3, 4, 5, 6, 7, 8, 9, 12, 13 \}$
$x > 5$	$ \{0, 2, 7, 10, 11, 13, 14 \}$
$2 < x < 10$	$ \{ 1, 6, 7, 12, 13 \}$
$5 < x < 10$	$ \{ 7, 13 \}$

Or represented as bitmaps which can be iterated over we would have the following:

value	$< 3$	$< 10$	$> 5$
10	0	0	1
3	0	1	0
15	0	0	1
0	1	1	0
0	1	1	0
1	1	1	0
5	0	1	0
6	0	1	1
2	1	1	0
1	1	1	0
12	0	0	1
14	0	0	1
3	0	1	0
9	0	1	1
11	0	0	1

Representing the array indexes as bitmaps is beneficial because they can be combined with bitmaps obtained from other indexes to satisfy complex filter constraints without needing indexes which themselves can understand every filter constraint. The following filters should be pushed down into a range index for efficiency reasons, but demonstrate how to build a double bounded range from two single bounded ranges:

value	$< 10$	$> 5$	$5 < x < 10$
10	0	1	0 & 1 = 0
3	1	0	1 & 0 = 0
15	0	1	0 & 1 = 0
0	1	0	1 & 0 = 0
0	1	0	1 & 0 = 0
1	1	0	1 & 0 = 0
5	1	0	1 & 0 = 0
6	1	1	1 & 1 = 1
2	1	0	1 & 0 = 0
1	1	0	1 & 0 = 0
12	0	1	0 & 1 = 0
14	0	1	0 & 1 = 0
3	1	0	1 & 0 = 0
9	1	1	1 & 1 = 1
11	0	1	0 & 1 = 0

Pinot’s indexes use RoaringBitmap to represent filters, so the range index will produce a RoaringBitmap too.

Bit slicing

The algorithm used to do range evaluations is described in the 1998 paper Bitmap Index Design and Evaluation. This algorithm has been used at least twice before: the paper mentions that SybaseIQ used it in the 90s, and pilosa also implemented it in the last decade. To understand it, first look at the binary layout of the numbers in the example, which have all been chosen to be representable in 4 bits to limit the number of columns I need to write.

value	bit 3	bit 2	bit 1	bit 0
10	1	0	1	0
3	0	0	1	1
15	1	1	1	1
0	0	0	0	0
0	0	0	0	0
1	0	0	0	1
5	0	1	0	1
6	0	1	1	0
2	0	0	1	0
1	0	0	0	1
12	1	1	0	0
14	1	1	1	0
3	0	0	1	1
9	1	0	0	1
11	1	0	1	1

The columns can be used to evaluate filters. For instance, the numbers greater than seven all have a bit in the leftmost column. Looking at the leftmost column would be enough to evaluate $x > 7$ but not $x > 8$ which also requires at least one bit in the three columns to the right. This gets coarser as the number of bits in the numbers increases so trying to post-filter wouldn’t scale well.

To arrive at an algorithm, each column can be range encoded so that there are two slices for each column: a bit is set in the $i$th slice if the bit is less than or equal to $i \in \{0, 1\}$.

value	predicate	bit 3	bit 2	bit 1	bit 0
10	$\leq 0$	0	1	0	1
	$\leq 1$	1	1	1	1
3	$\leq 0$	1	1	0	0
	$\leq 1$	1	1	1	1
15	$\leq 0$	0	0	0	0
	$\leq 1$	1	1	1	1
0	$\leq 0$	1	1	1	1
	$\leq 1$	1	1	1	1
0	$\leq 0$	1	1	1	1
	$\leq 1$	1	1	1	1
1	$\leq 0$	1	1	1	0
	$\leq 1$	1	1	1	1
5	$\leq 0$	1	0	1	0
	$\leq 1$	1	1	1	1
6	$\leq 0$	1	0	0	1
	$\leq 1$	1	1	1	1
2	$\leq 0$	1	1	0	1
	$\leq 1$	1	1	1	1
1	$\leq 0$	1	1	1	0
	$\leq 1$	1	1	1	1
12	$\leq 0$	0	0	1	1
	$\leq 1$	1	1	1	1
14	$\leq 0$	0	0	0	1
	$\leq 1$	1	1	1	1
3	$\leq 0$	1	1	0	0
	$\leq 1$	1	1	1	1
9	$\leq 0$	0	1	1	0
	$\leq 1$	1	1	1	1
11	$\leq 0$	0	1	0	0
	$\leq 1$	1	1	1	1

Of course, a bit can only ever be $\leq 1$ because they only take two values, so the second slice is redundant and the table becomes:

value	bit 3	bit 2	bit 1	bit 0
10	0	1	0	1
3	1	1	0	0
15	0	0	0	0
0	1	1	1	1
0	1	1	1	1
1	1	1	1	0
5	1	0	1	0
6	1	0	0	1
2	1	1	0	1
1	1	1	1	0
12	0	0	1	1
14	0	0	0	1
3	1	1	0	0
9	0	1	1	0
11	0	1	0	0

This mapping is easy to perform because it’s just the logical complement of the input values masked by how many bits we care about, so x -> ~x & 0xF in this case.

Now there is a bitset in each slice:

0	1	2	3
`100110011011000`	`000111100110010`	`110111001100111`	`010111111100100`

Performing a transposition like this incurs no spatial overhead, and can end up smaller than storing the data itself if the values have many leading zeros allowing slices to be pruned, especially if bitmap compression is applied to the columns.

A predicate $x \leq t$ can be evaluated against the slices, assuming $n$ rows were indexed as follows:

Assume all rows match, initialise bitmap state to $[0, n)$, that is set every bit between 0 and $n$.
for each bit in t:
1. if t[i] is set, set state = state | slices[i]
  1. These bits need to be included in the result because they mean this coefficient in the input was less than the same coefficient of the threshold
2. if t[i] is not set, set state = state & slices[i]
  1. Bits absent in the slice need to be removed because it means the bit was present

To evaluate an $x > t$ predicate, just take the logical complement of the result of an $x \leq t$ predicate. Predicates $x < t$ or $x \geq t$ can be evaluated by adding or subtracting 1 from the threshold and evaluating a $x \leq t$ or $x > t$ predicate respectively. A double ended range $t \leq x \leq u$ can be evaluated by intersecting $x > t$ and $x \leq u$.

This algorithm has strengths and weaknesses. Firstly, it should be clear it can’t compete with the potentialities of sorting the data: binary search over run-length encoded values or even the raw data has lower time complexity. The complexity is linear in the product of the number of rows and the number of slices, so evaluation time depends on the largest value in the data set. However, logical operations between bitmaps are very efficient and can be vectorised, so hundreds of rows can be evaluated in a handful of CPU instructions. Space is saved by storing the row indexes implicitly in the positions of the bits (and this is improved by the compression explained later), and it automatically produces sorted outputs. The various tree data structures which could also solve this problem can’t compress the row indexes (though some can compress the values) and would all need to sort the row indexes after evaluating the predicate, which prevents lazy evaluation.

Generalisation to higher bases

Any number can be expressed as the coefficients of a polynomial with base $b$, binary numbers just have coefficients 0 or 1. Other bases can be chosen, which create space/time tradeoffs best visualised by considering the representations of numbers in different bases:

base	representation
2	111010110111100110100010101
3	22121022020212200
4	13112330310111
8	726746425
10	123456789
16	75bcd15

In base-2, a large number 123456789 has 27 digits, but only one slice per digit, so the range encoding operation is lightning fast and has no write-amplification. At the other end of the spectrum, in base-16 there are only 7 digits but 15 slices per digits. Reducing the number of digits reduces the number of bitmap operations which need to be performed, but increase spatial overhead and write amplification. Bitmap Index Design and Evaluation explores this tradeoff (see here for implementations).

Example

This is an example of how to evaluate the predicate $x \leq 9$ or $x < 10$ against a 4-bit base-2 index.
There is a table for each step of the algorithm:

The far left column in each has the value as encountered in the data, the next four columns represent the values stored in the index; these columns never change.
The next column represents the state bitmap, which changes on each step of the algorithm (and if it doesn’t, this step should be eliminated if it can be detected).
The final two columns represent the target, the bitmap we know we want to have produced once the algorithm has terminated, and whether the state bitmap is currently accurate.
- The far right column should give the intuition that the algorithm doesn’t converge to a solution and randomly flips bits depending on what’s in the slice. In this example, the hamming distance between the state and the solution only increases until the last step.
The bold column is the slice being operated on.

The threshold 9 in binary is 1001, so the first and last slices will be united, and the middle slices will be intersected.

Initialise state

value	slice 3	slice 2	slice 1	slice 0	state	target
10	0	1	0	1	1	0	❌
3	1	1	0	0	1	1	✔️
15	0	0	0	0	1	0	❌
0	1	1	1	1	1	1	✔️
0	1	1	1	1	1	1	✔️
1	1	1	1	0	1	1	✔️
5	1	0	1	0	1	1	✔️
6	1	0	0	1	1	1	✔️
2	1	1	0	1	1	1	✔️
1	1	1	1	0	1	1	✔️
12	0	0	1	1	1	0	❌
14	0	0	0	1	1	0	❌
3	1	1	0	0	1	1	✔️
9	0	1	1	0	1	1	✔️
11	0	1	0	0	1	0	❌

Union with slice 0

Perform a destructive logical union between the first slice and the state, modifying the state bitmap.

value	slice 3	slice 2	slice 1	slice 0	state	target
10	0	1	0	1	1	0	❌
3	1	1	0	0	1	1	✔️
15	0	0	0	0	1	0	❌
0	1	1	1	1	1	1	✔️
0	1	1	1	1	1	1	✔️
1	1	1	1	0	1	1	✔️
5	1	0	1	0	1	1	✔️
6	1	0	0	1	1	1	✔️
2	1	1	0	1	1	1	✔️
1	1	1	1	0	1	1	✔️
12	0	0	1	1	1	0	❌
14	0	0	0	1	1	0	❌
3	1	1	0	0	1	1	✔️
9	0	1	1	0	1	1	✔️
11	0	1	0	0	1	0	❌

This step didn’t actually change the state bitmap, and could have been eliminated by checking if the LSB is set.

Intersection with slice 1

Perform a destructive logical intersection between the second slice and the state bitmap, modifying the state bitmap.

value	slice 3	slice 2	slice 1	slice 0	state	target
10	0	1	0	1	0	0	✔️
3	1	1	0	0	0	1	❌
15	0	0	0	0	0	0	✔️
0	1	1	1	1	1	1	✔️
0	1	1	1	1	1	1	✔️
5	1	0	1	0	1	1	✔️
1	1	1	1	0	1	1	✔️
6	1	0	0	1	0	1	❌
2	1	1	0	1	0	1	❌
1	1	1	1	0	1	1	✔️
12	0	0	1	1	1	0	❌
14	0	0	0	1	0	0	✔️
3	1	1	0	0	0	1	❌
9	0	1	1	0	1	1	❌
11	0	1	0	0	0	0	✔️

Intersection with slice 2

Perform a destructive logical intersection between the third slice and the state bitmap, modifying the state bitmap.

value	slice 3	slice 2	slice 1	slice 0	state	target
10	0	1	0	1	0	0	✔️
3	1	1	0	0	0	1	❌
15	0	0	0	0	0	0	✔️
0	1	1	1	1	1	1	✔️
0	1	1	1	1	1	1	✔️
5	1	0	1	0	0	1	❌
1	1	1	1	0	1	1	✔️
6	1	0	0	1	0	1	❌
2	1	1	0	1	0	1	❌
1	1	1	1	0	1	1	✔️
12	0	0	1	1	0	0	✔️
14	0	0	0	1	0	0	✔️
3	1	1	0	0	0	1	❌
9	0	1	1	0	1	1	✔️
11	0	1	0	0	0	0	✔️

Union with slice 3 and terminate

Perform a destructive logical union between the fourth slice and the state bitmap, modifying the state bitmap.

value	slice 3	slice 2	slice 1	slice 0	state	target
10	0	1	0	1	0	0	✔️
3	1	1	0	0	1	1	✔️
15	0	0	0	0	0	0	✔️
0	1	1	1	1	1	1	✔️
0	1	1	1	1	1	1	✔️
5	1	0	1	0	1	1	✔️
1	1	1	1	0	1	1	✔️
6	1	0	0	1	1	1	✔️
2	1	1	0	1	1	1	✔️
1	1	1	1	0	1	1	✔️
12	0	0	1	1	0	0	✔️
14	0	0	0	1	0	0	✔️
3	1	1	0	0	1	1	✔️
9	0	1	1	0	1	1	✔️
11	0	1	0	0	0	0	✔️

There are no more slices so the algorithm terminates here.

Optimisations

The algorithm’s time complexity is super-linear, depending on both the number of indexed rows and how many slices are present in the indexed values. This means that every slice which doesn’t need to be operated on (the first step in the example above was redundant) should be eliminated.

There are various fast-forwarding optimisations which eliminate bitmap operations between slices at the start of the evaluation. Some of these depend on the threshold:

Evaluating the result for the first slice has redundancy: if the LSB is set in the threshold, the first union is redundant. Similarly, if it isn’t set, the state bitmap can be initialised to the values of the first slice; initialising to [0, max) beforehand is redundant.
If there is a run of $k$ set bits starting from 0, all $k$ operations can be eliminated.

Others depend on the data:

If there is an empty slice at position $k$ and the bit $k$ is absent from the threshold $x$, the state bitmap will be empty afterwards. All $k$ operations can be replaced by just setting state to $\emptyset$.
If there is a full slice at position $k$ and the bit $k$ is present in the threshold $x$, the state bitmap will be full afterwards, so all $k$ operations can be replaced by setting state to $[0, max)$

The example above was for a 4-bit index, but arbitrary size numbers (64 bits is an artificial limit) can be indexed in theory, given a choice of a maximum value. However, allocating capacity for values much larger than the largest indexed value leads to unnecessary slices and slower evaluations. Knowing the maximum value reduces the number of slices which need to be operated on.

Similarly, if the minimum value is large, subtracting it from each value reduces the number of slices by a factor of $\log_2(x_{min})$. This is especially important for indexing timestamps, where the minimum timestamp in a data set will often have a large offset from 1970-01-01. At the time of writing, the unix epoch time is $1646510472$, and a collection of timestamps over the next 24 hours would fall in the range $[1646510472, 1646596872]$. The number of slices required to encode this range would be $ \lceil \log_2( 1646596872 ) \rceil = 31$, whereas to index $[0, 1646596872 - 1646510472]$ or $[0, 86400]$ only $ \lceil \log_2(86400) \rceil = 17$ slices are required, which roughly halves the evaluation time.

Horizontal and Vertical Evaluations

An important implementation choice is whether to operate on an entire slice at a time (vertically) or across a small horizontal section of all slices producing a section of the final result (horizontally).

In the step by step example of the algorithm above, the evaluation was vertical. Instead, let’s say we perform all four operations two rows at a time (recall that to evaluate $v \leq 9$ there was a union followed by two intersections and a final union) and append the partial result to an output incrementally. The evaluation would proceed as follows:

OR slice 0, AND slice 1, AND slice 2, OR slice 3, append result for rows 0, 1

value	slice 3	slice 2	slice 1	slice 0	state	target
10	0	1	0	1	0	0	✔️
3	1	1	0	0	1	1	✔️
15	0	0	0	0	-	0	❓
0	1	1	1	1	-	1	❓️
0	1	1	1	1	-	1	❓️
1	1	1	1	0	-	1	❓️

OR slice 0, AND slice 1, AND slice 2, OR slice 3, append result for rows 2, 3

value	slice 3	slice 2	slice 1	slice 0	state	target
10	0	1	0	1	-	0	✔️
3	1	1	0	0	-	1	✔️
15	0	0	0	0	0	0	✔️
0	1	1	1	1	1	1	✔️
0	1	1	1	1	-	1	❓️
1	1	1	1	0	-	1	❓️

In reality, hundreds or thousands of bits would be operated on at a time; two was just chosen to make it possible to visualise.

There are pros and cons either way. Owen Kaser and Daniel Lemire describe the trade-offs in Threshold and Symmetric Functions over Bitmaps:

“Implementations of bitmap operations can be described as “horizontal” or “vertical”. The former implementation produces its output incrementally, and can be stopped early. The latter implementation produces the entire completed bitmap. Horizontal approaches would be preferred if the output of the algorithm is to be consumed immediately, as we may be able to avoid storing the entire bitmap. As well, the producer and consumer can run in parallel. Vertical implementations may be simpler and have less overhead. If the output of the algorithm needs to be stored because it is consumed more than once, there may be no substantial space advantage to a horizontal implementation. With uncompressed bitmaps, a particularly horizontal implementation exists: a word of each input is read, after which one word of the output is produced.”

There are good reasons to prefer a horizontal approach beyond the ability to iterate over the result before it has been computed.

Cache efficiency: operating on slices horizontally means that no horizontal section of a slice ever needs to be loaded twice. Intermediate state for computing a horizontal section of the result can stay in cache if it is reused.
Density: if we have an arbitrary collection of values, there is no good reason to expect the slices to be sparse, unless the values are sorted. If the values were sorted, there are better choices of data structure. This means that the ability of compressed bitmaps like RoaringBitmap (see here or here) to exploit sparseness to prune intersections will be unreliable. If there are regions where some slices are sparse, so long as fast-forward optimisations are implemented, some work can be pruned anyway.

My benchmarks in the RoaringBitmap library showed that RangeBitmap usually outperforms a vertical implementation using RoaringBitmap by more than 2x.

RangeBitmap design

RangeBitmap in the RoaringBitmap Java library implements this algorithm. The data structure reuses some of RoaringBitmap’s building blocks, so here is a quick recap. RoaringBitmap is a compressed bitmap which prefix compresses integers, grouping values into aligned buckets of width $2^{16}$, storing the high 16 bits of each integer just once. There are different kinds of Container for storing the lower 16 bits, depending on the distribution of values within the bucket. If there are fewer than $2^{12}$ values in the bucket, an ArrayContainer - just a sorted array of 16 bit values - is used to store the values. Otherwise a BitmapContainer - an 8KB bitset - is used instead. However, if the values can be run-length encoded into fewer than $2^{11}$ runs, a RunContainer is used, which is just an array of 16-bit values, where the value at every even index is the starting position of a run, and the next value is the length of the run. The array is sorted by the starting positions of the runs.

Layout

RangeBitmap uses RoaringBitmap’s container implementations to represent the slices, but does away with prefix compression. This is based on a few heuristics which appear to hold for real-world datasets:

Lower order slices will be incompressible in most cases (the least significant slice could only ever be run-length encoded, and only if there are runs of even or odd values).
Unless the values are entirely random, higher order slices will be very dense or very sparse as values will tend to cluster.

This makes prefix compression useless for lower order slices because every row will have values here, and gives it a good chance of being useless in higher order slices depending on how the values cluster and whether the slices are full or empty. If every $2^{16}$ range is partially populated, the prefix can be represented implicitly by the container’s position. A RoaringBitmap of size $n$ has $ \lceil \frac{n}{2^{16}} \rceil $ 16 bit values to store integer prefixes. We should expect at least the bottom 5 slices (i.e. the values modulo 32) to have values in every $2^{16}$ range, so we can save at least $5 \times 2 \times \lceil \frac{n}{2^{16}} \rceil$ bytes by getting rid of the prefixes, even if all the higher order slices somehow end up empty or sparse. Getting rid of the prefixes and using the position to store the high 16 bits implicitly frees up space for metadata which can be used to determine whether a slice has any values for a $2^{16}$ range or not.

For each $2^{16}$ range, there is a mask which has the nth bit set if slice n is non-empty. These masks are stored contiguously, and if all slices are empty in a $2^{16}$ range, then an empty mask needs to be stored because the high 16 bits are implied by the position. The mask sizes are rounded up to the next multiple of 8 so that long sequences of leading zeros are not stored (if there are only 10 slices, a 64 bit mask would be wasteful, and a 16 bit mask would be used instead).

After the masks, RoaringBitmap containers are stored in ascending slice order. If a slice is empty for the $n$th slice, it is not stored, which makes random access to a $2^{16}$ range impossible - the containers must be iterated from the start.

There is a small header before the masks which helps with memory-mapping:

field	size (bytes)	purpose
cookie	2	allows checks whether a supplied buffer is not a `RangeBitmap`, as well as evolution
base	1	this allows evolution to e.g. base 4 or base 8 slicing
slice count	1	determines how many slices exist, which allows the size of the mask to
number of masks	2	this is actually redundant and could have been derived from the next field
max row	4	the index of the last value, e.g. 10M if 10M values are indexed

Construction and Memory Mapping

A RangeBitmap is immutable, and there are good reasons for this:

Slice pruning optimisations work better when the range of values is known ahead of time.
There’s not much point in supporting mutability without supporting concurrent updates, and there’s a lot of state to keep consistent, even though every update is an append. Fast query execution relies on the ability to vectorise operations, which would be hampered by supporting row level concurrent appends.
It just isn’t necessary for OLAP where mutable data structures are short-lived and immutable data structures are long-lived. It would be better to choose a data structure which simplifies concurrent updates at the expense of query efficiency for indexing live data.

A two phase construction process where values are first buffered and then sealed models this best:

var appender = RangeBitmap.appender(maxValue);
getLongValues().forEach(appender::add);
RangeBitmap bitmap = appender.build();

To write the data to disk so that it can be memory mapped (or otherwise loaded from disk into a ByteBuffer), the process is a little more convoluted:

var appender = RangeBitmap.appender(maxValue);
getLongValues().forEach(appender::add);
ByteBuffer buffer = allocateByteBuffer(appender.serializedSizeInBytes());
appender.serialize(buffer);

After executing the code above, the ByteBuffer will contain the header described above, the masks, and some RoaringBitmap containers laid out so that it can be queried without deserialization. A RangeBitmap can be loaded from a ByteBuffer in a matter of nanoseconds as follows:

var bitmap = RangeBitmap.map(buffer);

The usual caveats of memory mapping apply, but RangeBitmap doesn’t care how the bytes get from disk into the ByteBuffer.

Encoding

Values need to be encoded into longs before indexing, which allows any primitive type to be indexed with the same API. Moreover, RangeBitmap interprets long values in unsigned order, which simplifies slice layout. This makes using the data structure fairly user-unfriendly, but this was a suitable API for its use in Apache Pinot’s range index. For example, a double is mapped to an unsigned long so that ordering is preserved as follows:

public static long ordinalOf(double value) {
  if (value == Double.POSITIVE_INFINITY) {
    return 0xFFFFFFFFFFFFFFFFL;
  }
  if (value == Double.NEGATIVE_INFINITY || Double.isNaN(value)) {
    return 0;
  }
  long bits = Double.doubleToLongBits(value);
  // need negatives to come before positives
  if ((bits & Long.MIN_VALUE) == Long.MIN_VALUE) {
    // conflate 0/-0, or reverse order of negatives
    bits = bits == Long.MIN_VALUE ? Long.MIN_VALUE : ~bits;
  } else {
    // positives after negatives
    bits ^= Long.MIN_VALUE;
  }
  return bits;
}

Subtracting the minimum value from every other value, as is done in Apache Pinot’s range index for the sake of reducing the number of slices, automatically anchors the value range to zero and makes all values implicitly unsigned.

Querying a `RangeBitmap`

Querying a mapped RangeBitmap is straightforward, with the API described by the table:

operation	method
$ \{ x : x < t \} $	`RoaringBitmap lt(long threshold)`
$ \{ x : x < t \} \cap C$	`RoaringBitmap lt(long threshold, RoaringBitmap context)`
$ \{ x : x \leq t \} $	`RoaringBitmap lte(long threshold)`
$ \{ x : x \leq t \} \cap C$	`RoaringBitmap lte(long threshold, RoaringBitmap context)`
$ \{ x : x > t \} $	`RoaringBitmap gt(long threshold)`
$ \{ x : x > t \} \cap C$	`RoaringBitmap gt(long threshold, RoaringBitmap context)`
$ \{ x : x \geq t \}$	`RoaringBitmap gte(long threshold)`
$ \{ x : x \geq t \} ∩ C$	`RoaringBitmap gte(long threshold, RoaringBitmap context)`
$ \{ x : t \leq x \leq u \} $	`RoaringBitmap between(long min, long max)`
$ \{ x : t \leq x \leq u \} \cap C$	Currently unimplemented

Some methods support pushing a set intersection down into range predicate evaluation, which allows skipping over regions of rows not found in the set to be intersected with.

RangeBitmap rangeBitmap = createRangeBitmap();
RoaringBitmap context = ...
RoaringBitmap range = rangeBitmap.lt(threshold);
RoaringBitmap intersection = RoarinBitmap.and(context, range);

The following is equivalent to the code above, but what’s below is more efficient because it doesn’t need to materialise an intermediate bitmap and skips doing range evaluations where the intersection would definitely be empty:

RangeBitmap rangeBitmap = createRangeBitmap();
RoaringBitmap context = ...
RoaringBitmap intersection = rangeBitmap.lt(threshold, context);

Performance Evaluation

I was reluctant to add this section to the post because it can’t really be a fair comparison and is more of a sanity check. Pinot already had range indexes, and the goal of implementing RangeBitmap was to improve on the old implementation, and very early measurements indicated that RangeBitmap would improve performance by 5-10x, so alternatives were not evaluated exhaustively. I haven’t included every possible implementation in this comparison, but I feel it’s reasonably balanced because I demonstrate how much faster queries over sorted data are.

Various implementations of this interface are compared with values of different distributions, the size of the input is 76MB (10M longs) in each case:

public interface RangeEvaluator {

  RoaringBitmap between(long min, long max);

  int serializedSize();

}

The distributions are varied just to show that it actually matters, rather than systematically. The distributions all produce duplicate values and the size of the range depends on the distributions because the benchmark selects ranges based on values at statically defined ranks. This means that results are comparable given a choice of distribution, but it also demonstrates that your mileage may vary as peculiarities of data sets can be an important factor. For implementations which only work with sorted inputs, the data is sorted beforehand.

As a baseline, imagine you’re lucky enough to have sorted data and can perform an out-of-the-box binary search. Arrays.binarySearch doesn’t make guarantees about which index is produced when there are duplicates so a linear scan is needed to find it after yielding an index (though this could use, say, a quadratic probing algorithm).

public class BinarySearch implements RangeEvaluator {

  private final long[] data;

  public BinarySearch(long[] data) {
    this.data = data;
  }

  @Override
  public RoaringBitmap between(long min, long max) {
    int start = Arrays.binarySearch(data, min);
    int begin = start >= 0 ? start : -start - 1;
    while (begin - 1 >= 0 && data[begin - 1] == min) {
      begin--;
    }
    int end = Arrays.binarySearch(data, begin, data.length, max);
    int finish = end >= 0 ? end : -end - 1;
    while (finish + 1 < data.length && data[finish + 1] == max) {
      finish++;
    }
    return RoaringBitmap.bitmapOfRange(begin, finish + 1);
  }

  @Override
  public int serializedSize() {
    return 0;
  }

}

Whilst this has no spatial overhead, the linear search is problematic and it can perform quite badly when there are many duplicates (see $\exp(0.5)$).

To get around this, the data structure below is a naive approximation to Apache Pinot’s sorted index:

public class IntervalsEvaluator implements RangeEvaluator {

  private final long[] uniqueValues;
  private final int[] ranges;

  public IntervalsEvaluator(long[] values) {
    long[] unique = new long[16];
    int[] ranges = new int[16];
    int numRanges = 0;
    int start = 0;
    long current = values[0];
    for (int i = 1; i < values.length; i++) {
      long value = values[i];
      if (current != value) {
        if (numRanges == unique.length) {
          unique = Arrays.copyOf(unique, numRanges * 2);
          ranges = Arrays.copyOf(ranges, numRanges * 2);
        }
        unique[numRanges] = current;
        ranges[numRanges] = start;
        numRanges++;
        current = value;
        start = i;
      }
    }
    unique[numRanges] = current;
    ranges[numRanges] = start;
    numRanges++;
    this.uniqueValues = Arrays.copyOf(unique, numRanges);
    this.ranges = Arrays.copyOf(ranges, numRanges);
  }

  @Override
  public RoaringBitmap between(long min, long max) {
    int start = Arrays.binarySearch(uniqueValues, min);
    int begin = start >= 0 ? start : -start - 1;
    int end = Arrays.binarySearch(uniqueValues, begin, uniqueValues.length, max + 1);
    int finish = end >= 0 ? end : -end - 1;
    return RoaringBitmap.bitmapOfRange(ranges[start], ranges[finish]);
  }

  @Override
  public int serializedSize() {
    return uniqueValues.length * Long.BYTES + ranges.length * Integer.BYTES;
  }
}

This could be micro-optimised for locality, but it’s hard to beat for this problem, even though it has a spatial overhead of up to 1.5x the size of the input. Note that the $\exp(0.5)$ case is slower because the range is larger because there are lots of duplicates, but none of these distributions produce unique values, so the size is always smaller than the data. In fact, none of the approaches which support range queries over unsorted data will even get close to this.

The first implementation to support unsorted inputs is a natural extension of binary search: make a copy of the data, sort it, and store an array of the indexes where the value was stored in the unsorted array. This is included as a proxy for the way trees which sort the values but not the indexes which I never found justification to implement. These would also need to store and post-process the indexes.

public class SortedValuesUnsortedIndexesEvaluator implements RangeEvaluator {
  
  private final long[] sortedValues;
  private final int[] indexes;

  public SortedValuesUnsortedIndexesEvaluator(long[] data) {
    List<LongIntPair> pairs = IntStream.range(0, data.length)
        .mapToObj(i -> new LongIntPair(data[i], i))
        .sorted()
        .collect(Collectors.toList());
    sortedValues = pairs.stream().mapToLong(pair -> pair.value).toArray();
    indexes = pairs.stream().mapToInt(pair -> pair.index).toArray();
  }

  @Override
  public RoaringBitmap between(long min, long max) {
    int start = Arrays.binarySearch(sortedValues, min);
    int begin = start >= 0 ? start : -start - 1;
    while (begin - 1 >= 0 && sortedValues[begin - 1] == min) {
      begin--;
    }
    int end = Arrays.binarySearch(sortedValues, begin, sortedValues.length, max);
    int finish = end >= 0 ? end : -end - 1;
    while (finish + 1 < sortedValues.length && sortedValues[finish + 1] == max) {
      finish++;
    }
    RoaringBitmap result = new RoaringBitmap();
    for (int i = begin; i < finish; i++) {
      result.add(indexes[i]);
    }
    return result;
  }

  @Override
  public int serializedSize() {
    return sortedValues.length * Long.BYTES + indexes.length * Integer.BYTES;
  }

  private static final class LongIntPair implements Comparable<LongIntPair> {

    private final long value;
    private final int index;

    private LongIntPair(long value, int index) {
      this.value = value;
      this.index = index;
    }

    @Override
    public int compareTo(LongIntPair o) {
      return Long.compare(value, o.value);
    }

    @Override
    public boolean equals(Object o) {
      if (this == o) {
        return true;
      }
      if (o == null || getClass() != o.getClass()) {
        return false;
      }

      LongIntPair that = (LongIntPair) o;

      if (value != that.value) {
        return false;
      }
      return index == that.index;
    }

    @Override
    public int hashCode() {
      int result = (int) (value ^ (value >>> 32));
      result = 31 * result + index;
      return result;
    }
  }
}

This doesn’t perform very well, and storing it requires 1.5x the size of the data, despite finding the boundaries quickly when there are few duplicates.

Part of the problem is it’s not very efficient to build a RoaringBitmap from unsorted data, and maybe Pinot needing a RoaringBitmap is an artificial constraint. If building a RoaringBitmap weren’t the bottleneck, copying and sorting a section of the indexes array might be.

InvertedIndexEvaluator is a natural extension of IntervalsEvaluator, instead of storing the start of the range, it stores a bitmap of row indexes which have the value.

public class InvertedIndexEvaluator implements RangeEvaluator {

  private final long[] uniqueValues;
  private final RoaringBitmap[] bitmaps;
  private final int serializedSize;

  public InvertedIndexEvaluator(long[] values, long[] sortedValues) {
    long[] unique = new long[16];
    int numRanges = 0;
    long current = sortedValues[0];
    for (int i = 1; i < sortedValues.length; i++) {
      long value = sortedValues[i];
      if (current != value) {
        if (numRanges == unique.length) {
          unique = Arrays.copyOf(unique, numRanges * 2);
        }
        unique[numRanges] = current;
        numRanges++;
        current = value;
      }
    }
    unique[numRanges] = current;
    numRanges++;
    this.uniqueValues = Arrays.copyOf(unique, numRanges);
    RoaringBitmapWriter<RoaringBitmap>[] writers = new RoaringBitmapWriter[numRanges];
    Arrays.setAll(writers, i -> RoaringBitmapWriter.writer().get());
    for (int i = 0; i < values.length; i++) {
      writers[Arrays.binarySearch(uniqueValues, values[i])].add(i);
    }
    RoaringBitmap[] bitmaps = new RoaringBitmap[writers.length];
    Arrays.setAll(bitmaps, i -> writers[i].get());
    this.bitmaps = bitmaps;
    int ss = uniqueValues.length * 8 + bitmaps.length * 4;
    for (RoaringBitmap bitmap : bitmaps) {
      ss += bitmap.serializedSizeInBytes();
    }
    this.serializedSize = ss;
  }

  @Override
  public RoaringBitmap between(long min, long max) {
    int start = Arrays.binarySearch(uniqueValues, min);
    int begin = start >= 0 ? start : -start - 1;
    int end = Arrays.binarySearch(uniqueValues, begin, uniqueValues.length, max + 1);
    int finish = end >= 0 ? end : -end - 1;
    RoaringBitmap bitmap = bitmaps[begin].clone();
    for (int i = begin + 1; i <= finish & i < bitmaps.length; i++) {
      bitmap.or(bitmaps[i]);
    }
    return bitmap;
  }

  @Override
  public int serializedSize() {
    return serializedSize;
  }
}

This is a good option when there are a lot of duplicates, but needing to merge many tiny bitmaps to produce an output is the bottleneck.

The index is always smaller than the data:

Pinot’s old range index was quite similar to this, except the posting lists corresponded to buckets, and required a scan to remove false positives after filtering. By range encoding an inverted index (that is, when indexing a row with a value update all posting lists greater than the value too) range queries can be very fast, but the indexes end up huge and the consequent write amplification is unacceptable.

The simplest approach is a scan:

public class Scan implements RangeEvaluator {

  private final long[] data;

  public Scan(long[] data) {
    this.data = data;
  }

  @Override
  public RoaringBitmap between(long min, long max) {
    RoaringBitmapWriter<RoaringBitmap> writer = RoaringBitmapWriter.writer().get();
    for (int i = 0; i < data.length; i++) {
      if (data[i] >= min && data[i] <= max) {
        writer.add(i);
      }
    }
    return writer.get();
  }

  @Override
  public int serializedSize() {
    return 0;
  }
}

This has no spatial overhead but is very slow, though this could be done a lot faster by Java programs once the Vector API is available so will be a better option in the future.

Finally, RangeBitmap:

public class RangeBitmapEvaluator implements RangeEvaluator {

  private final RangeBitmap bitmap;
  private final int serializedSize;

  public RangeBitmapEvaluator(long[] data) {
    long min = Long.MAX_VALUE;
    long max = Long.MIN_VALUE;
    for (long datum : data) {
      min = Math.min(min, datum);
      max = Math.max(max, datum);
    }
    var appender = RangeBitmap.appender(max);
    for (long datum : data) {
      appender.add(datum - min);
    }
    serializedSize = appender.serializedSizeInBytes();
    this.bitmap = appender.build();
  }

  @Override
  public RoaringBitmap between(long min, long max) {
    return bitmap.between(min, max);
  }

  @Override
  public int serializedSize() {
    return serializedSize;
  }
}

It doesn’t get close to IntervalsEvaluator, but it’s the least bad considered option for unsorted values, it is always smaller than the data.

Excluding the implementations which only work on sorted inputs, RangeBitmap looks like the least bad option. It is only beaten by the inverted index in one case, is an order of magnitude faster than scanning or the sorted-values/unsorted-indexes approach and the inverted index in other cases.

It also has the least spatial overhead except for scanning which has none.

The source code for these benchmarks is located here

Usage in Apache Pinot

Apache Pinot allows range indexing of immutable segments, and since 0.9.0 these have been implemented in terms of RangeBitmap, enabled via a feature flag. From 0.10.0, this feature flag is enabled by default, and just needs the configuration in the documentation here. Users who have enabled the feature flag (which I won’t name: just upgrade to 0.10.0) have reported improvements of upwards of 5x on Pinot’s old range index implementation.

Performance Myths and Continuous Profiling

2021-12-27T00:00:00+00:00

My last post was about five very simple things you can do to avoid Java programs from being slower than they need to be. The reception to this post was mixed. Some readers agreed that the problems mentioned in the post were indeed very common, while others suggested more common inefficient patterns to avoid. For example, I could have suggested to precompile regular expressions, or not to program by exception, or to avoid String.format, but I felt this was all well covered already. The aim of the post was to help Java programmers to program defensively for efficiency - just as many do for correctness - by giving an idea of what some common things cost. However, several readers dismissed the content of the post as premature optimisation.

It’s interesting how negative responses to posts about minor software efficiencies can be. There will be various causes for this reaction, but a common theme is the existence of performance myths perpetuated by bad blog posts. I certainly hope that I’m not perpetuating performance myths but this position is understandable.

Performance myths can be split into two categories, and I’ll give an example of each:

Those that were never true
Those that used to be true

I think it’s the second category which cause the strongest reactions because they counteract future improvements. In this post I take a look at examples of both before discussing continuous profiling since the items I chose were based on things I’ve seen show up in profiles a lot.

Never true: prefer pre-increment to post-increment for loop induction variables
True once: prefer StringBuilder to string concatenation
Continuous profiling

Never true: prefer pre-increment to post-increment for loop induction variables

I have encountered claims that loops which use pre-increment for loop induction variables are faster than those which use post-increment. So:

for (int i = 0; i < size; i++) {
   doIt();  
}

would be slightly slower than:

for (int i = 0; i < size; ++i) {
   doIt();  
}

Even if this were true, the loop body would dominate the cost of control flow in most cases. It’s worth looking at the origin of the myth and the reasoning behind it though. The myth comes from C++, where it’s possible to overload increment operators. As you are probably aware, post-increment yields the value before performing the increment. This may lead to copying the value, so unless the compiler eliminates the copy because it’s not observable, pre-increment would have an advantage. I’m not going to dig in to how often or when this is true in C++ because it has no relevance to programs written in Java.

Since the JIT compiler optimises loops, it’s very unlikely that it will generate code corresponding to either i++ or ++i. This can be seen by analysing a simple benchmark:

@State(Scope.Benchmark)
public class Increments {

  @Param("1024")
  int size;

  private int[] input;
  private int[] output;

  @Setup(Level.Trial)
  public void setup() {
    input = ThreadLocalRandom.current().ints(size).toArray();
    output = new int[size];
  }

  @Benchmark
  public void autovecPre(Blackhole bh) {
    for (int i = 0; i < input.length; ++i) {
      output[i] += input[i];
    }
    bh.consume(output);
  }

  @Benchmark
  public void autovecPost(Blackhole bh) {
    for (int i = 0; i < input.length; i++) {
      output[i] += input[i];
    }
    bh.consume(output);
  }

  @Benchmark
  public int reducePre(Blackhole bh) {
    int sum = 0;
    for (int i = 0; i < input.length; ++i) {
      sum += Integer.bitCount(input[i]);
    }
    return sum;
  }

  @Benchmark
  public int reducePost(Blackhole bh) {
    int sum = 0;
    for (int i = 0; i < input.length; i++) {
      sum += Integer.bitCount(input[i]);
    }
    return sum;
  }

  @Benchmark
  public void blackholedPre(Blackhole bh) {
    for (int i = 0; i < input.length; ++i) {
      bh.consume(i);
    }
  }

  @Benchmark
  public void blackholedPost(Blackhole bh) {
    for (int i = 0; i < input.length; i++) {
      bh.consume(i);
    }
  }
}

The benchmark consists of a pre- and post-increment version of three loops selected carefully. Loops autovecPre and autovecPost can be autovectorised. Loops reducePre and reducePost can’t be autovectorised (because the operation in the loop body can’t, but fortunately this will no longer be the case on some hardware, see JDK-8278868) but can be unrolled. Using Blackhold::consume prevents loop unrolling, so different code will be generated in blackholedPre and blackholedPost than in the other loops.

Firstly, there is no difference in the scores for these benchmarks (pre is slower once but it’s just noise):

Benchmark                  (size)  Mode  Cnt     Score   Error  Units
Increments.autovecPost       1024  avgt    5   156.744 ± 0.416  ns/op
Increments.autovecPre        1024  avgt    5   158.390 ± 0.096  ns/op
Increments.blackholedPost    1024  avgt    5  2798.470 ± 2.657  ns/op
Increments.blackholedPre     1024  avgt    5  2798.259 ± 2.553  ns/op
Increments.reducePost        1024  avgt    5   456.012 ± 0.267  ns/op
Increments.reducePre         1024  avgt    5   455.922 ± 0.299  ns/op

This is because the loops are JIT-compiled exactly the same way.

Here’s autovecPre and autovecPost, where i is incremented by 64 at a time: add $0x40,%edx.

....[Hottest Region 1]..............................................................................
c2, level 4, inc.generated.Increments_autovecPost_jmhTest::autovecPost_avgt_jmhStub, version 472 (464 bytes) 

                       0x00007fd6f435aff4: vzeroupper 
                       0x00007fd6f435aff7: add    $0x30,%rsp
                       0x00007fd6f435affb: pop    %rbp
                       0x00007fd6f435affc: mov    0x108(%r15),%r10
                       0x00007fd6f435b003: test   %eax,(%r10)        
                       0x00007fd6f435b006: retq   
                       0x00007fd6f435b007: nop                       
                                                                     
                                                                     
         ↗         ↗   0x00007fd6f435b008: vmovdqu 0x10(%r8,%rdx,4),%ymm0
  0.73%  │         │   0x00007fd6f435b00f: vpaddd 0x10(%r10,%rdx,4),%ymm0,%ymm0
  0.75%  │         │   0x00007fd6f435b016: vmovdqu %ymm0,0x10(%r8,%rdx,4)
         │         │                                                 
         │         │                                                 
         │         │   0x00007fd6f435b01d: add    $0x8,%edx          
         │         │                                                 
         │         │                                                 
  0.66%  │         │   0x00007fd6f435b020: cmp    %r9d,%edx
         ╰         │   0x00007fd6f435b023: jl     0x00007fd6f435b008 
                   │                                                 
                   │                                                 
                 ↗ │↗  0x00007fd6f435b025: cmp    %r11d,%edx
          ╭      │ ││  0x00007fd6f435b028: jge    0x00007fd6f435b03d
  0.02%   │      │ ││  0x00007fd6f435b02a: xchg   %ax,%ax            
          │      │ ││                                                
          │      │ ││                                                
          │↗     │ ││  0x00007fd6f435b02c: mov    0x10(%r10,%rdx,4),%ebx
  0.54%   ││     │ ││  0x00007fd6f435b031: add    %ebx,0x10(%r8,%rdx,4)
          ││     │ ││                                                
          ││     │ ││                                                
  0.58%   ││     │ ││  0x00007fd6f435b036: inc    %edx               
          ││     │ ││                                                
          ││     │ ││                                                
  0.54%   ││     │ ││  0x00007fd6f435b038: cmp    %r11d,%edx
          │╰     │ ││  0x00007fd6f435b03b: jl     0x00007fd6f435b02c 
          │      │ ││                                                
          │      │ ││                                                
          ↘ ↗    │ ││  0x00007fd6f435b03d: mov    %rcx,%rdx
  0.23%     │    │ ││  0x00007fd6f435b040: shl    $0x3,%rdx          
            │    │ ││                                                
            │    │ ││                                                
            │    │ ││  0x00007fd6f435b044: mov    0x10(%rsp),%rsi
  0.10%     │    │ ││  0x00007fd6f435b049: data16 xchg %ax,%ax
            │    │ ││  0x00007fd6f435b04c: vzeroupper 
  0.29%     │    │ ││  0x00007fd6f435b04f: callq  0x00007fd6ec89cb00 
            │    │ ││                                                
            │    │ ││                                                
            │    │ ││                                                
            │    │ ││                                                
            │    │ ││  0x00007fd6f435b054: mov    0x40(%rsp),%r10
  0.17%     │    │ ││  0x00007fd6f435b059: movzbl 0x94(%r10),%r10d   
            │    │ ││                                                
            │    │ ││                                                
  0.10%     │    │ ││  0x00007fd6f435b061: mov    0x108(%r15),%r11
            │    │ ││  0x00007fd6f435b068: add    $0x1,%rbp          
            │    │ ││                                                
            │    │ ││                                                
  0.08%     │    │ ││  0x00007fd6f435b06c: test   %eax,(%r11)        
            │    │ ││  0x00007fd6f435b06f: test   %r10d,%r10d
            │    │ ││  0x00007fd6f435b072: jne    0x00007fd6f435afcf 
            │    │ ││                                                
  0.02%     │    │ ││  0x00007fd6f435b078: mov    0x50(%rsp),%r10
            │    │ ││  0x00007fd6f435b07d: mov    0x10(%r10),%r10d   
            │    │ ││                                                
            │    │ ││                                                
  0.12%     │    │ ││  0x00007fd6f435b081: mov    0xc(%r12,%r10,8),%r11d  
            │    │ ││                                                
            │    │ ││                                                
            │    │ ││                                                
            │    │ ││  0x00007fd6f435b086: mov    0x50(%rsp),%r8
  0.06%     │    │ ││  0x00007fd6f435b08b: mov    0x14(%r8),%ecx     
            │    │ ││                                                
            │    │ ││                                                
            │    │ ││  0x00007fd6f435b08f: test   %r11d,%r11d
            ╰    │ ││  0x00007fd6f435b092: jbe    0x00007fd6f435b03d  
                 │ ││                                                
                 │ ││                                                
  0.23%          │ ││  0x00007fd6f435b094: mov    0xc(%r12,%rcx,8),%r8d  
                 │ ││                                                
                 │ ││                                                
                 │ ││                                                
                 │ ││  0x00007fd6f435b099: test   %r8d,%r8d
             ╭   │ ││  0x00007fd6f435b09c: jbe    0x00007fd6f435b1ed
  0.08%      │   │ ││  0x00007fd6f435b0a2: mov    %r11d,%r9d
             │   │ ││  0x00007fd6f435b0a5: dec    %r9d
  0.17%      │   │ ││  0x00007fd6f435b0a8: cmp    %r8d,%r9d
             │╭  │ ││  0x00007fd6f435b0ab: jae    0x00007fd6f435b1ed
             ││  │ ││  0x00007fd6f435b0b1: cmp    %r11d,%r9d
             ││╭ │ ││  0x00007fd6f435b0b4: jae    0x00007fd6f435b1ed
  0.08%      │││ │ ││  0x00007fd6f435b0ba: shl    $0x3,%r10
             │││ │ ││  0x00007fd6f435b0be: lea    (%r12,%rcx,8),%r8
  0.19%      │││ │ ││  0x00007fd6f435b0c2: mov    %r8d,%r9d
             │││ │ ││  0x00007fd6f435b0c5: shr    $0x2,%r9d
  0.06%      │││ │ ││  0x00007fd6f435b0c9: and    $0x7,%r9d
             │││ │ ││  0x00007fd6f435b0cd: mov    $0x3,%ebx
  0.19%      │││ │ ││  0x00007fd6f435b0d2: sub    %r9d,%ebx
             │││ │ ││  0x00007fd6f435b0d5: and    $0x7,%ebx
  0.06%      │││ │ ││  0x00007fd6f435b0d8: inc    %ebx
             │││ │ ││  0x00007fd6f435b0da: cmp    %r11d,%ebx
  0.08%      │││ │ ││  0x00007fd6f435b0dd: cmovg  %r11d,%ebx
             │││ │ ││  0x00007fd6f435b0e1: xor    %edx,%edx          
             │││ │ ││                                                
             │││ │ ││                                                
  0.48%      │││↗│ ││  0x00007fd6f435b0e3: mov    0x10(%r10,%rdx,4),%edi
             │││││ ││  0x00007fd6f435b0e8: add    %edi,0x10(%r8,%rdx,4)  
             │││││ ││                                                
             │││││ ││                                                
  0.95%      │││││ ││  0x00007fd6f435b0ed: inc    %edx               
             │││││ ││                                                
             │││││ ││                                                
             │││││ ││  0x00007fd6f435b0ef: cmp    %ebx,%edx
             │││╰│ ││  0x00007fd6f435b0f1: jl     0x00007fd6f435b0e3  
             │││ │ ││                                                
             │││ │ ││                                                
  0.02%      │││ │ ││  0x00007fd6f435b0f3: mov    %r11d,%ebx
             │││ │ ││  0x00007fd6f435b0f6: add    $0xffffffc1,%ebx
  0.21%      │││ │ ││  0x00007fd6f435b0f9: cmp    %ebx,%edx
             │││ ╰ ││  0x00007fd6f435b0fb: jge    0x00007fd6f435b025  
             │││   ││                                                
             │││   ││                                                
  2.90%      │││  ↗││  0x00007fd6f435b101: vmovdqu 0x10(%r8,%rdx,4),%ymm0
  0.04%      │││  │││  0x00007fd6f435b108: vpaddd 0x10(%r10,%rdx,4),%ymm0,%ymm0
 10.08%      │││  │││  0x00007fd6f435b10f: vmovdqu %ymm0,0x10(%r8,%rdx,4)
  1.00%      │││  │││  0x00007fd6f435b116: vmovdqu 0x30(%r8,%rdx,4),%ymm0
             │││  │││  0x00007fd6f435b11d: vpaddd 0x30(%r10,%rdx,4),%ymm0,%ymm0
  9.27%      │││  │││  0x00007fd6f435b124: vmovdqu %ymm0,0x30(%r8,%rdx,4)
  0.79%      │││  │││  0x00007fd6f435b12b: vmovdqu 0x50(%r8,%rdx,4),%ymm0
             │││  │││  0x00007fd6f435b132: vpaddd 0x50(%r10,%rdx,4),%ymm0,%ymm0
  8.07%      │││  │││  0x00007fd6f435b139: vmovdqu %ymm0,0x50(%r8,%rdx,4)
  0.89%      │││  │││  0x00007fd6f435b140: vmovdqu 0x70(%r8,%rdx,4),%ymm0
             │││  │││  0x00007fd6f435b147: vpaddd 0x70(%r10,%rdx,4),%ymm0,%ymm0
  9.58%      │││  │││  0x00007fd6f435b14e: vmovdqu %ymm0,0x70(%r8,%rdx,4)
  0.52%      │││  │││  0x00007fd6f435b155: vmovdqu 0x90(%r8,%rdx,4),%ymm0
             │││  │││  0x00007fd6f435b15f: vpaddd 0x90(%r10,%rdx,4),%ymm0,%ymm0
  8.34%      │││  │││  0x00007fd6f435b169: vmovdqu %ymm0,0x90(%r8,%rdx,4)
  0.64%      │││  │││  0x00007fd6f435b173: vmovdqu 0xb0(%r8,%rdx,4),%ymm0
             │││  │││  0x00007fd6f435b17d: vpaddd 0xb0(%r10,%rdx,4),%ymm0,%ymm0
  9.71%      │││  │││  0x00007fd6f435b187: vmovdqu %ymm0,0xb0(%r8,%rdx,4)
  0.58%      │││  │││  0x00007fd6f435b191: vmovdqu 0xd0(%r8,%rdx,4),%ymm0
             │││  │││  0x00007fd6f435b19b: vpaddd 0xd0(%r10,%rdx,4),%ymm0,%ymm0
 14.68%      │││  │││  0x00007fd6f435b1a5: vmovdqu %ymm0,0xd0(%r8,%rdx,4)
  0.54%      │││  │││  0x00007fd6f435b1af: vmovdqu 0xf0(%r8,%rdx,4),%ymm0
             │││  │││  0x00007fd6f435b1b9: vpaddd 0xf0(%r10,%rdx,4),%ymm0,%ymm0
  9.46%      │││  │││  0x00007fd6f435b1c3: vmovdqu %ymm0,0xf0(%r8,%rdx,4)  
             │││  │││                                                
             │││  │││                                                
  0.52%      │││  │││  0x00007fd6f435b1cd: add    $0x40,%edx         
             │││  │││                                                
             │││  │││                                                
             │││  │││  0x00007fd6f435b1d0: cmp    %ebx,%edx
             │││  ╰││  0x00007fd6f435b1d2: jl     0x00007fd6f435b101
  0.23%      │││   ││  0x00007fd6f435b1d8: mov    %r11d,%r9d
             │││   ││  0x00007fd6f435b1db: add    $0xfffffff9,%r9d
  0.10%      │││   ││  0x00007fd6f435b1df: cmp    %r9d,%edx
             │││   ╰│  0x00007fd6f435b1e2: jl     0x00007fd6f435b008
             │││    ╰  0x00007fd6f435b1e8: jmpq   0x00007fd6f435b025 
             │││                                                     
             │││                                                     
             ↘↘↘       0x00007fd6f435b1ed: mov    $0xffffff7e,%esi
                       0x00007fd6f435b1f2: mov    %r11d,0x18(%rsp)
                       0x00007fd6f435b1f7: nop
                       0x00007fd6f435b1f8: vzeroupper 
                       0x00007fd6f435b1fb: callq  0x00007fd6ec89d280  
                                                                     
....................................................................................................
 95.73%  <total for region 1>

....[Hottest Region 1]..............................................................................
c2, level 4, inc.generated.Increments_autovecPre_jmhTest::autovecPre_avgt_jmhStub, version 460 (464 bytes) 
             
                       0x00007f44cc35d2f4: vzeroupper 
                       0x00007f44cc35d2f7: add    $0x30,%rsp
                       0x00007f44cc35d2fb: pop    %rbp
                       0x00007f44cc35d2fc: mov    0x108(%r15),%r10
                       0x00007f44cc35d303: test   %eax,(%r10)        
                       0x00007f44cc35d306: retq   
                       0x00007f44cc35d307: nop                       
                                                                     
                                                                     
  0.77%  ↗         ↗   0x00007f44cc35d308: vmovdqu 0x10(%r8,%rdx,4),%ymm0
  0.25%  │         │   0x00007f44cc35d30f: vpaddd 0x10(%r10,%rdx,4),%ymm0,%ymm0
  3.27%  │         │   0x00007f44cc35d316: vmovdqu %ymm0,0x10(%r8,%rdx,4)
         │         │                                                 
         │         │                                                 
  0.29%  │         │   0x00007f44cc35d31d: add    $0x8,%edx          
         │         │                                                 
         │         │                                                 
  0.10%  │         │   0x00007f44cc35d320: cmp    %r9d,%edx
         ╰         │   0x00007f44cc35d323: jl     0x00007f44cc35d308  
                   │                                                 
                   │                                                 
  0.19%          ↗ │↗  0x00007f44cc35d325: cmp    %r11d,%edx
          ╭      │ ││  0x00007f44cc35d328: jge    0x00007f44cc35d33d
          │      │ ││  0x00007f44cc35d32a: xchg   %ax,%ax            
          │      │ ││                                                
          │      │ ││                                                
  0.50%   │↗     │ ││  0x00007f44cc35d32c: mov    0x10(%r10,%rdx,4),%ebx
          ││     │ ││  0x00007f44cc35d331: add    %ebx,0x10(%r8,%rdx,4)
          ││     │ ││                                                
          ││     │ ││                                                
  1.06%   ││     │ ││  0x00007f44cc35d336: inc    %edx               
          ││     │ ││                                                
          ││     │ ││                                                
          ││     │ ││  0x00007f44cc35d338: cmp    %r11d,%edx
          │╰     │ ││  0x00007f44cc35d33b: jl     0x00007f44cc35d32c 
          │      │ ││                                                
          │      │ ││                                                
  0.10%   ↘ ↗    │ ││  0x00007f44cc35d33d: mov    %rcx,%rdx
            │    │ ││  0x00007f44cc35d340: shl    $0x3,%rdx          
            │    │ ││                                                
            │    │ ││                                                
  0.17%     │    │ ││  0x00007f44cc35d344: mov    0x10(%rsp),%rsi
            │    │ ││  0x00007f44cc35d349: data16 xchg %ax,%ax
  0.10%     │    │ ││  0x00007f44cc35d34c: vzeroupper 
  0.10%     │    │ ││  0x00007f44cc35d34f: callq  0x00007f44c489cb00 
            │    │ ││                                                
            │    │ ││                                               
            │    │ ││                                               
            │    │ ││                                               
  0.02%     │    │ ││  0x00007f44cc35d354: mov    0x40(%rsp),%r10
  0.08%     │    │ ││  0x00007f44cc35d359: movzbl 0x94(%r10),%r10d   
            │    │ ││                                                
            │    │ ││                                                
            │    │ ││  0x00007f44cc35d361: mov    0x108(%r15),%r11
  0.04%     │    │ ││  0x00007f44cc35d368: add    $0x1,%rbp          
            │    │ ││                                                
            │    │ ││                                                
            │    │ ││  0x00007f44cc35d36c: test   %eax,(%r11)        
  0.25%     │    │ ││  0x00007f44cc35d36f: test   %r10d,%r10d
            │    │ ││  0x00007f44cc35d372: jne    0x00007f44cc35d2cf 
            │    │ ││                                                
            │    │ ││  0x00007f44cc35d378: mov    0x50(%rsp),%r10
  0.04%     │    │ ││  0x00007f44cc35d37d: mov    0x10(%r10),%r10d   
            │    │ ││                                                
            │    │ ││                                                
            │    │ ││  0x00007f44cc35d381: mov    0xc(%r12,%r10,8),%r11d
            │    │ ││                                                
            │    │ ││                                                
            │    │ ││                                                
  0.15%     │    │ ││  0x00007f44cc35d386: mov    0x50(%rsp),%r8
            │    │ ││  0x00007f44cc35d38b: mov    0x14(%r8),%ecx     
            │    │ ││                                                
            │    │ ││                                                
  0.04%     │    │ ││  0x00007f44cc35d38f: test   %r11d,%r11d
            ╰    │ ││  0x00007f44cc35d392: jbe    0x00007f44cc35d33d 
                 │ ││                                                
                 │ ││                                                
  0.06%          │ ││  0x00007f44cc35d394: mov    0xc(%r12,%rcx,8),%r8d
                 │ ││                                                
                 │ ││                                                
                 │ ││                                                
  0.27%          │ ││  0x00007f44cc35d399: test   %r8d,%r8d
             ╭   │ ││  0x00007f44cc35d39c: jbe    0x00007f44cc35d4ed
  0.15%      │   │ ││  0x00007f44cc35d3a2: mov    %r11d,%r9d
             │   │ ││  0x00007f44cc35d3a5: dec    %r9d
  0.17%      │   │ ││  0x00007f44cc35d3a8: cmp    %r8d,%r9d
             │╭  │ ││  0x00007f44cc35d3ab: jae    0x00007f44cc35d4ed
  0.08%      ││  │ ││  0x00007f44cc35d3b1: cmp    %r11d,%r9d
             ││╭ │ ││  0x00007f44cc35d3b4: jae    0x00007f44cc35d4ed
  0.19%      │││ │ ││  0x00007f44cc35d3ba: shl    $0x3,%r10
             │││ │ ││  0x00007f44cc35d3be: lea    (%r12,%rcx,8),%r8
             │││ │ ││  0x00007f44cc35d3c2: mov    %r8d,%r9d
  0.12%      │││ │ ││  0x00007f44cc35d3c5: shr    $0x2,%r9d
  0.04%      │││ │ ││  0x00007f44cc35d3c9: and    $0x7,%r9d
             │││ │ ││  0x00007f44cc35d3cd: mov    $0x3,%ebx
             │││ │ ││  0x00007f44cc35d3d2: sub    %r9d,%ebx
  0.10%      │││ │ ││  0x00007f44cc35d3d5: and    $0x7,%ebx
  0.17%      │││ │ ││  0x00007f44cc35d3d8: inc    %ebx
             │││ │ ││  0x00007f44cc35d3da: cmp    %r11d,%ebx
             │││ │ ││  0x00007f44cc35d3dd: cmovg  %r11d,%ebx
  0.06%      │││ │ ││  0x00007f44cc35d3e1: xor    %edx,%edx          
             │││ │ ││                                                
             │││ │ ││                                                
  0.19%      │││↗│ ││  0x00007f44cc35d3e3: mov    0x10(%r10,%rdx,4),%edi
  0.21%      │││││ ││  0x00007f44cc35d3e8: add    %edi,0x10(%r8,%rdx,4)
             │││││ ││                                                
             │││││ ││                                                
  0.50%      │││││ ││  0x00007f44cc35d3ed: inc    %edx               
             │││││ ││                                                
             │││││ ││                                                
  0.21%      │││││ ││  0x00007f44cc35d3ef: cmp    %ebx,%edx
             │││╰│ ││  0x00007f44cc35d3f1: jl     0x00007f44cc35d3e3 
             │││ │ ││                                                
             │││ │ ││                                                
  0.08%      │││ │ ││  0x00007f44cc35d3f3: mov    %r11d,%ebx
             │││ │ ││  0x00007f44cc35d3f6: add    $0xffffffc1,%ebx
             │││ │ ││  0x00007f44cc35d3f9: cmp    %ebx,%edx
             │││ ╰ ││  0x00007f44cc35d3fb: jge    0x00007f44cc35d325  
             │││   ││                                                
             │││   ││                                                
  3.25%      │││  ↗││  0x00007f44cc35d401: vmovdqu 0x10(%r8,%rdx,4),%ymm0
  0.42%      │││  │││  0x00007f44cc35d408: vpaddd 0x10(%r10,%rdx,4),%ymm0,%ymm0
  9.16%      │││  │││  0x00007f44cc35d40f: vmovdqu %ymm0,0x10(%r8,%rdx,4)
  1.29%      │││  │││  0x00007f44cc35d416: vmovdqu 0x30(%r8,%rdx,4),%ymm0
             │││  │││  0x00007f44cc35d41d: vpaddd 0x30(%r10,%rdx,4),%ymm0,%ymm0
  8.47%      │││  │││  0x00007f44cc35d424: vmovdqu %ymm0,0x30(%r8,%rdx,4)
  0.60%      │││  │││  0x00007f44cc35d42b: vmovdqu 0x50(%r8,%rdx,4),%ymm0
             │││  │││  0x00007f44cc35d432: vpaddd 0x50(%r10,%rdx,4),%ymm0,%ymm0
 13.43%      │││  │││  0x00007f44cc35d439: vmovdqu %ymm0,0x50(%r8,%rdx,4)
  1.04%      │││  │││  0x00007f44cc35d440: vmovdqu 0x70(%r8,%rdx,4),%ymm0
             │││  │││  0x00007f44cc35d447: vpaddd 0x70(%r10,%rdx,4),%ymm0,%ymm0
  7.91%      │││  │││  0x00007f44cc35d44e: vmovdqu %ymm0,0x70(%r8,%rdx,4)
  0.58%      │││  │││  0x00007f44cc35d455: vmovdqu 0x90(%r8,%rdx,4),%ymm0
             │││  │││  0x00007f44cc35d45f: vpaddd 0x90(%r10,%rdx,4),%ymm0,%ymm0
  9.66%      │││  │││  0x00007f44cc35d469: vmovdqu %ymm0,0x90(%r8,%rdx,4)
  0.58%      │││  │││  0x00007f44cc35d473: vmovdqu 0xb0(%r8,%rdx,4),%ymm0
             │││  │││  0x00007f44cc35d47d: vpaddd 0xb0(%r10,%rdx,4),%ymm0,%ymm0
  9.14%      │││  │││  0x00007f44cc35d487: vmovdqu %ymm0,0xb0(%r8,%rdx,4)
  0.50%      │││  │││  0x00007f44cc35d491: vmovdqu 0xd0(%r8,%rdx,4),%ymm0
             │││  │││  0x00007f44cc35d49b: vpaddd 0xd0(%r10,%rdx,4),%ymm0,%ymm0
  9.08%      │││  │││  0x00007f44cc35d4a5: vmovdqu %ymm0,0xd0(%r8,%rdx,4)
  0.83%      │││  │││  0x00007f44cc35d4af: vmovdqu 0xf0(%r8,%rdx,4),%ymm0
             │││  │││  0x00007f44cc35d4b9: vpaddd 0xf0(%r10,%rdx,4),%ymm0,%ymm0
  8.52%      │││  │││  0x00007f44cc35d4c3: vmovdqu %ymm0,0xf0(%r8,%rdx,4)
             │││  │││                                                
             │││  │││                                                
  0.37%      │││  │││  0x00007f44cc35d4cd: add    $0x40,%edx         
             │││  │││                                                
             │││  │││                                                
             │││  │││  0x00007f44cc35d4d0: cmp    %ebx,%edx
             │││  ╰││  0x00007f44cc35d4d2: jl     0x00007f44cc35d401
  0.15%      │││   ││  0x00007f44cc35d4d8: mov    %r11d,%r9d
             │││   ││  0x00007f44cc35d4db: add    $0xfffffff9,%r9d
             │││   ││  0x00007f44cc35d4df: cmp    %r9d,%edx
             │││   ╰│  0x00007f44cc35d4e2: jl     0x00007f44cc35d308
             │││    ╰  0x00007f44cc35d4e8: jmpq   0x00007f44cc35d325 
             │││                                                     
             │││                                                     
             ↘↘↘       0x00007f44cc35d4ed: mov    $0xffffff7e,%esi
                       0x00007f44cc35d4f2: mov    %r11d,0x18(%rsp)
                       0x00007f44cc35d4f7: nop
                       0x00007f44cc35d4f8: vzeroupper 
....................................................................................................
 95.13%  <total for region 1>

I haven’t included the perfasm output for the other loops, but the compiled code is also identical. In reducePre and reducePost the loop is unrolled so i is incremented 8 at a time (add $0x8,%esi). In the loops contrived not to be unrolled, the increments are performed one at a time using inc %ebp, but the code is still identical.

Even though this is a myth, does it actually cause any harm? Of course not, loops optimised under this delusion are compiled the same way. The only negative scenario would be if a misguided team member were to insist on this practice being followed.

True once: prefer StringBuilder to string concatenation

A very long time ago, in JDK1.4, when concatenating Strings, a StringBuffer was used.

public class StringExample {
  
  public static String concatenate(String left, String right) {
    return left + right;
  }

  public static String concatenateStringBuilder(String left, String right) {
    return new StringBuilder(left.length() + right.length())
            .append(left).append(right).toString();
  }
}

The top method above would use StringBuffer, which has synchronized methods:

  public static java.lang.String concatenate(java.lang.String, java.lang.String);
    Code:
       0: new           #2                  // class java/lang/StringBuffer
       3: dup
       4: invokespecial #3                  // Method java/lang/StringBuffer."<init>":()V
       7: aload_0
       8: invokevirtual #4                  // Method java/lang/StringBuffer.append:(Ljava/lang/String;)Ljava/lang/StringBuffer;
      11: aload_1
      12: invokevirtual #4                  // Method java/lang/StringBuffer.append:(Ljava/lang/String;)Ljava/lang/StringBuffer;
      15: invokevirtual #5                  // Method java/lang/StringBuffer.toString:()Ljava/lang/String;
      18: areturn

  public static java.lang.String concatenateStringBuilder(java.lang.String, java.lang.String);
    Code:
       0: new           #6                  // class java/lang/StringBuilder
       3: dup
       4: aload_0
       5: invokevirtual #7                  // Method java/lang/String.length:()I
       8: aload_1
       9: invokevirtual #7                  // Method java/lang/String.length:()I
      12: iadd
      13: invokespecial #8                  // Method java/lang/StringBuilder."<init>":(I)V
      16: aload_0
      17: invokevirtual #9                  // Method java/lang/StringBuilder.append:(Ljava/lang/String;)Ljava/lang/StringBuilder;
      20: aload_1
      21: invokevirtual #9                  // Method java/lang/StringBuilder.append:(Ljava/lang/String;)Ljava/lang/StringBuilder;
      24: invokevirtual #10                 // Method java/lang/StringBuilder.toString:()Ljava/lang/String;

This meant that using a StringBuilder at language level 1.4 would be beneficial. If you learnt Java when 1.4 was commonly used, you might think this is an essential optimisation.

Later, the generated bytecode was improved, so that StringBuilder would be used instead. In JDK1.8 (and earlier) the bytecode looks like this:

  public static java.lang.String concatenate(java.lang.String, java.lang.String);
    Code:
       0: new           #2                  // class java/lang/StringBuilder
       3: dup
       4: invokespecial #3                  // Method java/lang/StringBuilder."<init>":()V
       7: aload_0
       8: invokevirtual #4                  // Method java/lang/StringBuilder.append:(Ljava/lang/String;)Ljava/lang/StringBuilder;
      11: aload_1
      12: invokevirtual #4                  // Method java/lang/StringBuilder.append:(Ljava/lang/String;)Ljava/lang/StringBuilder;
      15: invokevirtual #5                  // Method java/lang/StringBuilder.toString:()Ljava/lang/String;
      18: areturn

  public static java.lang.String concatenateStringBuilder(java.lang.String, java.lang.String);
    Code:
       0: new           #2                  // class java/lang/StringBuilder
       3: dup
       4: aload_0
       5: invokevirtual #6                  // Method java/lang/String.length:()I
       8: aload_1
       9: invokevirtual #6                  // Method java/lang/String.length:()I
      12: iadd
      13: invokespecial #7                  // Method java/lang/StringBuilder."<init>":(I)V
      16: aload_0
      17: invokevirtual #4                  // Method java/lang/StringBuilder.append:(Ljava/lang/String;)Ljava/lang/StringBuilder;
      20: aload_1
      21: invokevirtual #4                  // Method java/lang/StringBuilder.append:(Ljava/lang/String;)Ljava/lang/StringBuilder;
      24: invokevirtual #5                  // Method java/lang/StringBuilder.toString:()Ljava/lang/String;

Maybe there’s a tiny benefit in sizing the StringBuilder, but now the code is harder to read for marginal gain. In JDK9, String concatenation was reworked, so the StringBuilder code is at a disadvantage to the more readable concatenation code:

  public static java.lang.String concatenate(java.lang.String, java.lang.String);
    Code:
       0: aload_0
       1: aload_1
       2: invokedynamic #2,  0              // InvokeDynamic #0:makeConcatWithConstants:(Ljava/lang/String;Ljava/lang/String;)Ljava/lang/String;
       7: areturn

  public static java.lang.String concatenateStringBuilder(java.lang.String, java.lang.String);
    Code:
       0: new           #3                  // class java/lang/StringBuilder
       3: dup
       4: aload_0
       5: invokevirtual #4                  // Method java/lang/String.length:()I
       8: aload_1
       9: invokevirtual #4                  // Method java/lang/String.length:()I
      12: iadd
      13: invokespecial #5                  // Method java/lang/StringBuilder."<init>":(I)V
      16: aload_0
      17: invokevirtual #6                  // Method java/lang/StringBuilder.append:(Ljava/lang/String;)Ljava/lang/StringBuilder;
      20: aload_1
      21: invokevirtual #6                  // Method java/lang/StringBuilder.append:(Ljava/lang/String;)Ljava/lang/StringBuilder;
      24: invokevirtual #7                  // Method java/lang/StringBuilder.toString:()Ljava/lang/String;

The invokedynamic bytecode instruction, which can be used to drive better optimisations, is used instead of the StringBuilder. The confusing thing is this only applies when the number of strings is known, so not in loops, but in most cases it’s now better just to use +. I won’t bother to measure this progress because the people who made these changes to javac did that at the time, but the point is that even the bytecode can change in such a way to render old optimisations neutral or negative.

In my last post, I tried to avoid putting forward items which might become untrue in the future, but didn’t quite manage this. I predict that four of the five items will stand the test of time, but item three (avoid iterating over Enum.values()) may eventually become a myth if frozen arrays become a reality in the future.

Continuous profiling

What these myths have in common is they focus on low level technical trivia rather than fundamental algorithmic or engineering constraints, and trivial details are as likely to be misunderstood as they are to change. There are more important factors to system efficiency than low-level programming details. For instance, being required by specification to do something wasteful by an external system is a problem which should be solved architecturally. If architectural blunders create bottlenecks, it’s very costly to fix.

Assuming a sensible architecture, the causes of many costs are neither dictated by nor are observable to external systems, so can be changed easily. Efficiency at this level depends on choices of algorithm, data structure, and level of mechanical sympathy. These kinds of problems can be surfaced by using continuous profiling, and are often surprising. Continuous profiling, by its nature, tends to highlight what’s important to reducing overall cost, and filters out distracting and obvious fixed costs in favour of unexpected unit costs. My impression is that many engineers believe profiling to be too expensive to run in production, but I think these engineers’ minds would be blown by how much CPU time is spent collecting logs, metrics, and traces if they tried it. The reality is that it costs about 3% CPU utilisation and will lead to double-digit savings which translate to significantly reduced running costs for all but the leanest applications.

I think a diagram, drawn by Aleksey Shipilёv several years ago, explains where continuous profiling is useful:

I relax by drawing stuff on my large whiteboard. Here's "Perf Work Phase Diagram" for you. pic.twitter.com/RnITiLoNFH
— Aleksey Shipilëv (@shipilev) March 18, 2015

Essentially, continuous profiling will help you get from an Amazingly stupid program to Beauty Peak by making it obvious where the costs are; the majority of programs needn’t go beyond Beauty Peak. This process doesn’t consist entirely of making the kinds of mundane point substitutions in my last post. It’s often necessary to take a step back and ask why something costly is happening at all, and rework the algorithm, rather than think about cost reduction.

Despite all this, once you have data about what your program spends its time doing, many of the culprits will be mundane unless you have a very interesting program. When there’s no increase in complexity incurred in using a more efficient idiom, there’s no need to wait for its necessity to be proven by a profiler. In any case, continuous profiling should help calibrate your sense of which idioms are costly, and avoid the myths.

5 Mundane Java Performance Tips

2021-11-28T00:00:00+00:00

Most of the time it isn’t really necessary to optimise software, but this post contains 5 tips to avoid making software written in Java slower for the sake of it.

How to read this post: it didn’t take long for the term premature optimisation to arise after posting this on the Internet. Apart from item 5 (you are burning money if you are still using JDK8), none of this advice is important enough to contort your code to take advantage of. The particular block of code may never get called much, which is where the mantra to fix performance problems once they have been identified comes from. However, for one reason or another, I have looked at probably thousands of profiles of Java code I didn’t write during my career, and all of these things have shown up in profiles several times. That is, these things do cause excessive resource consumption in real applications and libraries. Sometimes, when there is a bottleneck, it indicates a design problem, and preventing the bottleneck from being called so often is usually more effective than speeding up the block of code at the bottleneck. On the other hand, you don’t need to be resizing HashMaps at least once per request if you have the information to size them properly in hand. If you can, size the HashMap properly, and it will never become a bottleneck showing up in other people’s profiles down the line.

Size HashMaps whenever possible
Use wrappers for composite HashMap keys
Don’t iterate over Enum.values()
Use enums instead of constant Strings
Stop using JDK8

Size HashMaps whenever possible

Even if most of its operations are quite fast, resizing HashMaps is slow and hard to optimise, so the size should be calculated before building the map. The size parameter does not take the load factor in to account, so the number of elements needs to be divided by the load factor, which is 0.75 by default: multiplying by 4/3 is usually enough.

There are four scenarios in the following benchmark: 10 and 14 keys are inserted into maps with default capacity (16) and maps with a capacity of 24 respectively.

@State(Scope.Benchmark)
public class HashMapResize {
  
    @Param({"10", "14"})
    int keys;

    @Param({"16", "24"})
    int capacity;
    
    @Benchmark
    public HashMap<Integer, Integer> loadHashMap() {
      HashMap<Integer, Integer> map = new HashMap<>(capacity);
      for (int i = 0; i < keys; ++i) {
        map.put(i, i);
      }
      return map;
    }
}

The maps have the default load factor of 0.75, so up to 12 keys can be inserted into a map with capacity 16 before a resize. To insert 14 keys without a resize would require an initial capacity of at least 19, but capacities are rounded up to the next power of 2, so 14 keys would end up in a map with a table of capacity 32. Since the initial capacity is rounded up to the next power of 2, it just so happens that when initial capacity 24 is requested, 24 keys would actually fit without a resize.

Benchmark	Mode	Threads	Samples	Score	Score Error (99.9%)	Unit	Param: capacity	Param: keys
HashMapResize.loadHashMap	avgt	1	5	191.729672	9.744065	ns/op	16	10
HashMapResize.loadHashMap:·gc.alloc.rate.norm	avgt	1	5	448.000079	0.000010	B/op	16	10
HashMapResize.loadHashMap	avgt	1	5	330.267466	14.739395	ns/op	16	14
HashMapResize.loadHashMap:·gc.alloc.rate.norm	avgt	1	5	720.000139	0.000025	B/op	16	14
HashMapResize.loadHashMap	avgt	1	5	221.798264	97.696476	ns/op	24	10
HashMapResize.loadHashMap:·gc.alloc.rate.norm	avgt	1	5	512.000092	0.000038	B/op	24	10
HashMapResize.loadHashMap	avgt	1	5	292.934060	13.489139	ns/op	24	14
HashMapResize.loadHashMap:·gc.alloc.rate.norm	avgt	1	5	640.000121	0.000016	B/op	24	14

It takes 70% longer to put 14 keys into a HashMap with capacity 16 than it does to insert 10 keys, and allocation rate per constructed map is 60% higher. Increasing the capacity to 24 saves 11% in allocation rate and 34% in build time per HashMap by avoiding a resize in this benchmark. Exact numbers will vary but the point is that, if the size of the map can be calculated easily, doing so will save time and reduce allocation rate.

Use wrappers for composite HashMap keys

Whenever a HashMap has composite String keys, use a wrapper instead of concatenating the strings to make a key. Doing so will make the lookup much faster and reduce allocation rate, as the benchmark below demonstrates. Two maps, one with concatenated keys and the other with keys in a Pair object are constructed so lookup time can be compared.

@State(Scope.Benchmark)
public class CompositeLookup {

    @Param("1024")
    int size;

    Map<String, Object> concatMap;
    Map<Pair, Object> pairMap;
    String[] prefixes;
    String[] suffixes;

    @Setup(Level.Trial)
    public void setup() {
      prefixes = new String[size];
      suffixes = new String[size];
      concatMap = new HashMap<>();
      pairMap = new HashMap<>();
      for (int i = 0; i < size; ++i) {
        prefixes[i] = UUID.randomUUID().toString();
        suffixes[i] = UUID.randomUUID().toString();
        concatMap.put(prefixes[i] + ";" + suffixes[i], i);
        // use new String to avoid reference equality speeding up the equals calls 
        pairMap.put(new Pair(new String(prefixes[i]), new String(suffixes[i])), i);
      }
    }

    @Benchmark
    @OperationsPerInvocation(1024)
    public void concatenate(Blackhole bh) { 
      for (int i = 0; i < prefixes.length; ++i) {
        bh.consume(concatMap.get(prefixes[i] + ";" + suffixes[i]));
      }
    }

    @Benchmark
    @OperationsPerInvocation(1024)
    public void wrap(Blackhole bh) {
      for (int i = 0; i < prefixes.length; ++i) {
        bh.consume(pairMap.get(new Pair(prefixes[i], suffixes[i])));
      }
    }
}

Benchmark	Mode	Threads	Samples	Score	Score Error (99.9%)	Unit	Param: size
CompositeLookup.concatenate	avgt	1	5	158.160452	5.622590	ns/op	1024
CompositeLookup.concatenate:·gc.alloc.rate.norm	avgt	1	5	120.000066	0.000010	B/op	1024
CompositeLookup.wrap	avgt	1	5	43.550144	0.932947	ns/op	1024
CompositeLookup.wrap:·gc.alloc.rate.norm	avgt	1	5	24.000018	0.000003	B/op	1024

Concatenating the keys takes 3.7x longer per lookup and allocates five times more (this depends on the size of the keys though). The reason wrapping outperforms concatenation is that a String instance caches its hash code, and constructing a new String requires calculation of a new hash code and String’s hash code algorithm also isn’t very efficient. The difference is actually huge, and I have seen this make a big difference in real applications many times - this idiom should be more common.

Vladimir Sitnikov points out that this pattern turned up as a bottleneck in Spring framework.

Don’t iterate over Enum.values()

An array is allocated every time Enum.values() is called, which can really add up. If you own the code, and the class won’t be visible to untrusted third parties (i.e. you are the end user), the best thing you can do is preallocate the array and use it instead of Enum.values(), on the basis that your own code won’t mutate it. Otherwise it can be stashed in a local variable wherever needed, which should alleviate concerns about mutability.

The benchmark below compares iteration over Enum.values(), a preallocated array populated from Enum.values() and over an EnumSet for enums of different sizes.

public class EnumIterationBenchmark {

  @Benchmark
  public void valuesFour(Blackhole bh) {
    for (Four it : Four.values()) {
      bh.consume(it.ordinal());
    }
  }

  @Benchmark
  public void valuesEight(Blackhole bh) {
    for (Eight it : Eight.values()) {
      bh.consume(it.ordinal());
    }
  }

  @Benchmark
  public void valuesSixteen(Blackhole bh) {
    for (Sixteen it : Sixteen.values()) {
      bh.consume(it.ordinal());
    }
  }

  @Benchmark
  public void cachedFour(Blackhole bh) {
    for (Four it : Four.VALUES) {
      bh.consume(it.ordinal());
    }
  }

  @Benchmark
  public void cachedEight(Blackhole bh) {
    for (Eight it : Eight.VALUES) {
      bh.consume(it.ordinal());
    }
  }

  @Benchmark
  public void cachedSixteen(Blackhole bh) {
    for (Sixteen it : Sixteen.VALUES) {
      bh.consume(it.ordinal());
    }
  }

  @Benchmark
  public void enumSetFour(Blackhole bh) {
    for (Four it : EnumSet.allOf(Four.class)) {
      bh.consume(it.ordinal());
    }
  }

  @Benchmark
  public void enumSetEight(Blackhole bh) {
    for (Eight it : EnumSet.allOf(Eight.class)) {
      bh.consume(it.ordinal());
    }
  }

  @Benchmark
  public void enumSetSixteen(Blackhole bh) {
    for (Sixteen it : EnumSet.allOf(Sixteen.class)) {
      bh.consume(it.ordinal());
    }
  }
}

Benchmark	Mode	Threads	Samples	Score	Score Error (99.9%)	Unit
EnumIterationBenchmark.cachedEight	avgt	1	5	28.644844	0.171543	ns/op
EnumIterationBenchmark.cachedEight:·gc.alloc.rate.norm	avgt	1	5	0.000012	0.000000	B/op
EnumIterationBenchmark.cachedFour	avgt	1	5	14.774474	0.072698	ns/op
EnumIterationBenchmark.cachedFour:·gc.alloc.rate.norm	avgt	1	5	0.000006	0.000000	B/op
EnumIterationBenchmark.cachedSixteen	avgt	1	5	56.613368	0.296960	ns/op
EnumIterationBenchmark.cachedSixteen:·gc.alloc.rate.norm	avgt	1	5	0.000023	0.000000	B/op
EnumIterationBenchmark.enumSetEight	avgt	1	5	53.622149	1.782352	ns/op
EnumIterationBenchmark.enumSetEight:·gc.alloc.rate.norm	avgt	1	5	32.000020	0.000001	B/op
EnumIterationBenchmark.enumSetFour	avgt	1	5	29.840317	1.830070	ns/op
EnumIterationBenchmark.enumSetFour:·gc.alloc.rate.norm	avgt	1	5	32.000011	0.000001	B/op
EnumIterationBenchmark.enumSetSixteen	avgt	1	5	105.247332	49.049506	ns/op
EnumIterationBenchmark.enumSetSixteen:·gc.alloc.rate.norm	avgt	1	5	32.000040	0.000018	B/op
EnumIterationBenchmark.valuesEight	avgt	1	5	33.237779	2.077222	ns/op
EnumIterationBenchmark.valuesEight:·gc.alloc.rate.norm	avgt	1	5	48.000014	0.000001	B/op
EnumIterationBenchmark.valuesFour	avgt	1	5	16.610465	0.906318	ns/op
EnumIterationBenchmark.valuesFour:·gc.alloc.rate.norm	avgt	1	5	32.000007	0.000000	B/op
EnumIterationBenchmark.valuesSixteen	avgt	1	5	56.557226	3.012126	ns/op
EnumIterationBenchmark.valuesSixteen:·gc.alloc.rate.norm	avgt	1	5	80.000023	0.000001	B/op

Calling Enum.values() will allocate an easily avoidable 80 bytes per call for a 16 element enum, though doesn’t pose a time penalty. EnumSets are tiny, so using them reduces allocation rate, but the iteration code is slower than iterating over an array so they don’t offer a good stand in.

I found a case of this which had existed in Spring framework for over a decade earlier in the year, where resolving HTTP status code to Spring’s HttpStatus enum was allocating ~1MB/s at very low request rate.

Use enums instead of constant Strings

There are obvious benefits to using enums instead of strings for constants because enums enforce validation, but they are generally good for performance: even though HashMap is fast, EnumMap is faster.

public class EnumMapBenchmark {

  @State(Scope.Benchmark)
  public static abstract class BaseState {
    @Param("10000")
    int size;

    @Param("42")
    int seed;

    int[] randomValues;

    @Setup(Level.Trial)
    public void setup() {
      SplittableRandom random = new SplittableRandom(seed);
      randomValues = new int[size];
      for (int i = 0; i < size; i++) {
        randomValues[i] = random.nextInt(0, Integer.MAX_VALUE);
      }
      fill(randomValues);
    }

    abstract void fill(int[] randomValues);
  }

  @State(Scope.Benchmark)
  public static class EnumMapState extends BaseState {

    EnumMap<AnEnum, String> map;
    AnEnum[] values;

    @Override
    void fill(int[] randomValues) {
      map = new EnumMap<>(AnEnum.class);
      values = new AnEnum[randomValues.length];
      AnEnum[] enumValues = AnEnum.values();
      int pos = 0;
      for (int i : randomValues) {
        values[pos++] = enumValues[i % enumValues.length];
      }
      for (AnEnum value : enumValues) {
        map.put(value, UUID.randomUUID().toString());
      }
    }
  }

  @State(Scope.Benchmark)
  public static class MixedState extends BaseState {

    EnumMap<AnEnum, String> map;
    String[] values;

    @Override
    void fill(int[] randomValues) {
      map = new EnumMap<>(AnEnum.class);
      values = new String[randomValues.length];
      AnEnum[] enumValues = AnEnum.values();
      int pos = 0;
      for (int i : randomValues) {
        values[pos++] = enumValues[i % enumValues.length].toString();
      }
      for (AnEnum value : enumValues) {
        map.put(value, UUID.randomUUID().toString());
      }
    }
  }

  @State(Scope.Benchmark)
  public static class HashMapState extends BaseState {

    HashMap<String, String> map;
    String[] values;

    @Override
    void fill(int[] randomValues) {
      map = new HashMap<>();
      values = new String[randomValues.length];
      AnEnum[] enumValues = AnEnum.values();
      int pos = 0;
      for (int i : randomValues) {
        values[pos++] = enumValues[i % enumValues.length].toString();
      }
      for (AnEnum value : enumValues) {
        map.put(value.toString(), UUID.randomUUID().toString());
      }
    }
  }

  @Benchmark
  public void enumMap(EnumMapState state, Blackhole bh) {
    for (AnEnum value : state.values) {
      bh.consume(state.map.get(value));
    }
  }

  @Benchmark
  public void hashMap(HashMapState state, Blackhole bh) {
    for (String value : state.values) {
      bh.consume(state.map.get(value));
    }
  }
}

Benchmark	Mode	Threads	Samples	Score	Score Error (99.9%)	Unit	Param: seed	Param: size
EnumMapBenchmark.enumMap	avgt	1	5	65.225800	7.990521	us/op	42	10000
EnumMapBenchmark.hashMap	avgt	1	5	151.394872	3.564463	us/op	42	10000

Naturally, there is a one time conversion cost to produce the enum (which ironically uses a HashMap) but once that’s been done it will pay for itself by allowing the use of EnumMap (and EnumSet) unless the value is basically inert. Don’t contort your code or risk being unable to consume data you don’t control just because EnumMap is fast, but if your data naturally fits an enum, then be sure to make use of EnumMap and EnumSet.

Stop using JDK8

All Java applications use a lot of Strings, and String was just a lot fatter and slower in JDK8. For example, consider constructing Strings from ASCII encoded byte[]s, which happens a lot if you do things like parse JSON or load classes.

@State(Scope.Benchmark)
public class UTF8Benchmark {

    @Param("UTF-8")
    String charsetName;

    @Param({"4", "20", "200", "2000"})
    int size;

    Charset charset;
    private byte[] bytes;
    private String string;

    @Setup(Level.Trial)
    public void setup() {
      this.charset = Charset.forName(charsetName);
      this.bytes = asciiBytes(size);
      this.string = new String(bytes, charset);
    }

    @Benchmark
    public String stringFromBytes() {
      return new String(bytes, charset);
    }
   
    @Benchmark
    public byte[] bytesFromString() {
      return string.getBytes(charset);
    }

    private static byte[] asciiByted(int size) {
      byte[] bytes = new byte[size];
      for (int i = 0; i < size; ++i) {
        bytes[i] = (byte) (i & 0x7F);
      }
      return bytes;
    }
}

The benchmark was run on JDK8 and JDK11. ASCII Strings (e.g. class names and metadata or JSON keys) were twice the size on JDK8 because the content was always stored in UTF-16, which explains why the allocation rate was halved in JDK11, but pay attention to how much faster the decoding is too.

JDK	Benchmark	Mode	Threads	Samples	Score	Score Error (99.9%)	Unit	Param: charsetName	Param: size
11	UTF8Benchmark.stringFromBytes	avgt	1	5	18.355847	0.674115	ns/op	UTF-8	4
8	UTF8Benchmark.stringFromBytes	avgt	1	5	51.693575	1.610212	ns/op	UTF-8	4
11	UTF8Benchmark.stringFromBytes:·gc.alloc.rate.norm	avgt	1	5	48.000008	0.000001	B/op	UTF-8	4
8	UTF8Benchmark.stringFromBytes:·gc.alloc.rate.norm	avgt	1	5	88.000023	0.000003	B/op	UTF-8	4
11	UTF8Benchmark.stringFromBytes	avgt	1	5	20.827067	3.695586	ns/op	UTF-8	20
8	UTF8Benchmark.stringFromBytes	avgt	1	5	60.232549	1.767569	ns/op	UTF-8	20
11	UTF8Benchmark.stringFromBytes:·gc.alloc.rate.norm	avgt	1	5	64.000009	0.000002	B/op	UTF-8	20
8	UTF8Benchmark.stringFromBytes:·gc.alloc.rate.norm	avgt	1	5	120.000026	0.000004	B/op	UTF-8	20
11	UTF8Benchmark.stringFromBytes	avgt	1	5	34.451863	9.095702	ns/op	UTF-8	200
8	UTF8Benchmark.stringFromBytes	avgt	1	5	159.427489	8.422851	ns/op	UTF-8	200
11	UTF8Benchmark.stringFromBytes:·gc.alloc.rate.norm	avgt	1	5	240.000014	0.000004	B/op	UTF-8	200
8	UTF8Benchmark.stringFromBytes:·gc.alloc.rate.norm	avgt	1	5	480.000070	0.000011	B/op	UTF-8	200
11	UTF8Benchmark.stringFromBytes	avgt	1	5	285.590238	75.077635	ns/op	UTF-8	2000
8	UTF8Benchmark.stringFromBytes	avgt	1	5	1350.229058	69.483579	ns/op	UTF-8	2000
11	UTF8Benchmark.stringFromBytes:·gc.alloc.rate.norm	avgt	1	5	2040.000118	0.000030	B/op	UTF-8	2000
8	UTF8Benchmark.stringFromBytes:·gc.alloc.rate.norm	avgt	1	5	4080.000601	0.000117	B/op	UTF-8	2000

It’s a similar story when encoding Strings to byte[] too, which is something your application will do if it does any kind of serialization, logging, or tracing.

JDK	Benchmark	Mode	Threads	Samples	Score	Score Error (99.9%)	Unit	Param: charsetName	Param: size
11	UTF8Benchmark.bytesFromString	avgt	1	5	13.835438	0.661028	ns/op	UTF-8	4
8	UTF8Benchmark.bytesFromString	avgt	1	5	62.958213	1.832734	ns/op	UTF-8	4
11	UTF8Benchmark.bytesFromString:·gc.alloc.rate.norm	avgt	1	5	24.000006	0.000001	B/op	UTF-8	4
8	UTF8Benchmark.bytesFromString:·gc.alloc.rate.norm	avgt	1	5	160.000028	0.000004	B/op	UTF-8	4
11	UTF8Benchmark.bytesFromString	avgt	1	5	17.457885	2.175969	ns/op	UTF-8	20
8	UTF8Benchmark.bytesFromString	avgt	1	5	75.760063	3.000610	ns/op	UTF-8	20
11	UTF8Benchmark.bytesFromString:·gc.alloc.rate.norm	avgt	1	5	40.000007	0.000002	B/op	UTF-8	20
8	UTF8Benchmark.bytesFromString:·gc.alloc.rate.norm	avgt	1	5	224.000033	0.000004	B/op	UTF-8	20
11	UTF8Benchmark.bytesFromString	avgt	1	5	30.758900	8.336043	ns/op	UTF-8	200
8	UTF8Benchmark.bytesFromString	avgt	1	5	227.269922	18.536450	ns/op	UTF-8	200
11	UTF8Benchmark.bytesFromString:·gc.alloc.rate.norm	avgt	1	5	216.000013	0.000003	B/op	UTF-8	200
8	UTF8Benchmark.bytesFromString:·gc.alloc.rate.norm	avgt	1	5	936.000100	0.000014	B/op	UTF-8	200
11	UTF8Benchmark.bytesFromString	avgt	1	5	280.886995	61.136388	ns/op	UTF-8	2000
8	UTF8Benchmark.bytesFromString	avgt	1	5	2138.703149	203.825640	ns/op	UTF-8	2000
11	UTF8Benchmark.bytesFromString:·gc.alloc.rate.norm	avgt	1	5	2016.000116	0.000024	B/op	UTF-8	2000
8	UTF8Benchmark.bytesFromString:·gc.alloc.rate.norm	avgt	1	5	8136.000937	0.000113	B/op	UTF-8	2000

These operations are so fundamental that it’s impossible to escape them.

Loop Fission

2021-11-27T00:00:00+00:00

Loop fission is a process normally applied by a compiler to loops to make them faster. The idea is to split larger loop bodies which perform several distinct tasks into separate loops, in the hope that the individual loops can be optimised more effectively in isolation. As far as I’m aware, C2, Hotspot’s JIT compiler, doesn’t do this so you have to do it yourself. I came across a couple of cases where this was profitable recently, and this post uses these as examples.

XOR
Group By

XOR

One of the operations which can be performed between two RoaringBitmap objects is an XOR or symmetric difference. After applying a destructive (i.e. one of the bitmaps will be mutated) XOR between two bitmaps, the mutated bitmap will contain a bit wherever the bitmaps differed before the XOR. One of the three container types in a RoaringBitmap is a BitmapContainer which contains a bitmap, a dense collection of bits. When a destructive XOR is applied between two BitmapContainers, the mutated container must have its bits updated, but they also need to be counted. This can be done in one loop or two separate loops, as in the representative benchmark below:

@State(Scope.Benchmark)
public class XorCount {

  @Param({"256", "512", "1024", "2048", "4096"})
  int size;

  private long[] left;
  private long[] right;

  @Setup(Level.Trial)
  public void setup() {
    left = new long[size];
    right = new long[size];
    for (int i = 0; i < size; i++) {
      left[i] = ThreadLocalRandom.current().nextLong();
      right[i] = ThreadLocalRandom.current().nextLong();
    }
  }


  @Benchmark
  public int fused() {
    int count = 0;
    for (int i = 0; i < left.length & i < right.length; i++) {
      left[i] ^= right[i];
      count += Long.bitCount(left[i]);
    }
    return count;
  }

  @Benchmark
  public int fissured() {
    for (int i = 0; i < left.length & i < right.length; i++) {
      left[i] ^= right[i];
    }
    int count = 0;
    for (long l : left) {
      count += Long.bitCount(l);
    }
    return count;
  }
}

It seems intuitive that fused would be faster; there is only one pass over the data, but this isn’t necessarily the case.

Benchmark	Mode	Threads	Samples	Score	Score Error (99.9%)	Unit	Param: size
XorCount.fissured	avgt	1	5	146.324870	0.166643	ns/op	256
XorCount.fissured	avgt	1	5	283.583423	0.156367	ns/op	512
XorCount.fissured	avgt	1	5	564.507047	0.512299	ns/op	1024
XorCount.fissured	avgt	1	5	1154.083245	6.005591	ns/op	2048
XorCount.fissured	avgt	1	5	2805.933619	10.880235	ns/op	4096
XorCount.fused	avgt	1	5	229.713707	0.138824	ns/op	256
XorCount.fused	avgt	1	5	462.435821	1.015399	ns/op	512
XorCount.fused	avgt	1	5	908.624781	0.759116	ns/op	1024
XorCount.fused	avgt	1	5	1802.717233	3.106054	ns/op	2048
XorCount.fused	avgt	1	5	3585.279127	6.716832	ns/op	4096

For small sizes, the fissured loop is noticeably faster, with convergence as the lengths grow. Eventually, the bitset would get so large that doing a single pass would make more sense as it makes better use of cache, but in a BitmapContainer the length is fixed to 1024 elements, so this is immaterial.

Why is the fissured loop faster? This is because of the loop which performs the XOR operation:

    for (int i = 0; i < left.length & i < right.length; i++) {
      left[i] ^= right[i];
    }

This loop can be autovectorised - many XORs can be computed in a single instruction. On my laptop, running JDK11 on Ubuntu, 256 bits or 4 longs are operated on from each array at a time.

  1.11%  │    │││ │↗      ││  0x00007f1bc835b892: vmovdqu 0x10(%rdi,%rdx,8),%ymm0
  0.31%  │    │││ ││      ││  0x00007f1bc835b898: vpxor  0x10(%r11,%rdx,8),%ymm0,%ymm0
  0.40%  │    │││ ││      ││  0x00007f1bc835b89f: vmovdqu %ymm0,0x10(%r11,%rdx,8)
  1.09%  │    │││ ││      ││  0x00007f1bc835b8a6: vmovdqu 0x30(%rdi,%rdx,8),%ymm0
  0.44%  │    │││ ││      ││  0x00007f1bc835b8ac: vpxor  0x30(%r11,%rdx,8),%ymm0,%ymm0
  1.69%  │    │││ ││      ││  0x00007f1bc835b8b3: vmovdqu %ymm0,0x30(%r11,%rdx,8)
  0.42%  │    │││ ││      ││  0x00007f1bc835b8ba: vmovdqu 0x50(%rdi,%rdx,8),%ymm0
  0.31%  │    │││ ││      ││  0x00007f1bc835b8c0: vpxor  0x50(%r11,%rdx,8),%ymm0,%ymm0
  0.77%  │    │││ ││      ││  0x00007f1bc835b8c7: vmovdqu %ymm0,0x50(%r11,%rdx,8)
  0.54%  │    │││ ││      ││  0x00007f1bc835b8ce: vmovdqu 0x70(%rdi,%rdx,8),%ymm0
  0.42%  │    │││ ││      ││  0x00007f1bc835b8d4: vpxor  0x70(%r11,%rdx,8),%ymm0,%ymm0
  2.28%  │    │││ ││      ││  0x00007f1bc835b8db: vmovdqu %ymm0,0x70(%r11,%rdx,8)
  0.33%  │    │││ ││      ││  0x00007f1bc835b8e2: vmovdqu 0x90(%rdi,%rdx,8),%ymm0
  0.46%  │    │││ ││      ││  0x00007f1bc835b8eb: vpxor  0x90(%r11,%rdx,8),%ymm0,%ymm0
  1.13%  │    │││ ││      ││  0x00007f1bc835b8f5: vmovdqu %ymm0,0x90(%r11,%rdx,8)
  0.19%  │    │││ ││      ││  0x00007f1bc835b8ff: vmovdqu 0xb0(%rdi,%rdx,8),%ymm0
  0.36%  │    │││ ││      ││  0x00007f1bc835b908: vpxor  0xb0(%r11,%rdx,8),%ymm0,%ymm0
  2.38%  │    │││ ││      ││  0x00007f1bc835b912: vmovdqu %ymm0,0xb0(%r11,%rdx,8)
  0.13%  │    │││ ││      ││  0x00007f1bc835b91c: vmovdqu 0xd0(%rdi,%rdx,8),%ymm0
  0.40%  │    │││ ││      ││  0x00007f1bc835b925: vpxor  0xd0(%r11,%rdx,8),%ymm0,%ymm0
  1.44%  │    │││ ││      ││  0x00007f1bc835b92f: vmovdqu %ymm0,0xd0(%r11,%rdx,8)
  0.15%  │    │││ ││      ││  0x00007f1bc835b939: vmovdqu 0xf0(%rdi,%rdx,8),%ymm0
  0.23%  │    │││ ││      ││  0x00007f1bc835b942: vpxor  0xf0(%r11,%rdx,8),%ymm0,%ymm0
  2.84%  │    │││ ││      ││  0x00007f1bc835b94c: vmovdqu %ymm0,0xf0(%r11,%rdx,8)

The other loop doesn’t get the same treatment from the compiler:

    int count = 0;
    for (long l : left) {
      count += Long.bitCount(l);
    }
    return count;

Each of the popcnt instructions below operate on 64 bits each.

  0.09%  │││              ↗   0x00007f3b1c35bd02: popcnt 0x28(%r11,%rbx,8),%r8
 12.50%  │││              │   0x00007f3b1c35bd09: popcnt 0x20(%r11,%rbx,8),%rdi
  0.02%  │││              │   0x00007f3b1c35bd10: popcnt 0x48(%r11,%rbx,8),%rdx
 12.86%  │││              │   0x00007f3b1c35bd17: popcnt 0x40(%r11,%rbx,8),%r9
  0.02%  │││              │   0x00007f3b1c35bd1e: popcnt 0x38(%r11,%rbx,8),%rax
  0.02%  │││              │   0x00007f3b1c35bd25: popcnt 0x30(%r11,%rbx,8),%rsi
         │││              │   0x00007f3b1c35bd2c: popcnt 0x18(%r11,%rbx,8),%r13
  7.61%  │││              │   0x00007f3b1c35bd33: popcnt 0x10(%r11,%rbx,8),%rbp
         │││              │   0x00007f3b1c35bd3a: add    %ecx,%ebp
  0.07%  │││              │   0x00007f3b1c35bd3c: add    %ebp,%r13d
  0.02%  │││              │   0x00007f3b1c35bd3f: add    %edi,%r13d
  8.25%  │││              │   0x00007f3b1c35bd42: add    %r8d,%r13d
  0.07%  │││              │   0x00007f3b1c35bd45: add    %r13d,%esi
  0.07%  │││              │   0x00007f3b1c35bd48: add    %esi,%eax
  7.94%  │││              │   0x00007f3b1c35bd4a: add    %eax,%r9d
  8.10%  │││              │   0x00007f3b1c35bd4d: add    %r9d,%edx

When the loops are fused (or aren’t fissured) the XORs aren’t vectorised and only operate on 64 bits at a time.

  0.31%    ↗│││   0x00007f6eac35baa0: mov    0x10(%r10,%r11,8),%rsi
  2.28%    ││││   0x00007f6eac35baa5: xor    0x10(%rbx,%r11,8),%rsi  
           ││││                                                 
           ││││                                                 
  7.69%    ││││   0x00007f6eac35baaa: mov    %rsi,0x10(%rbx,%r11,8)
           ││││                                                 
           ││││                                                 
  2.06%    ││││   0x00007f6eac35baaf: popcnt %rsi,%rsi
  8.92%    ││││   0x00007f6eac35bab4: add    %esi,%edx

In the fused loop, the loop advances at the pace of the slowest operation. At some point, the array would get so large that not doing two passes over the array would trump better code generation, which might be why C2 is cautious here. Notably, the popcount operation can be vectorised and clang does so, so the fused loop written in C++ and compiled with clang would be much faster. clang will also perform loop fission (which it calls distribution).
This is a good example of performance intuition from one language not carrying over into another.

Group By

I recently started working on Apache Pinot, which is a scalable OLAP store. In a group by query, values from one column are aggregated by values in another, and in Pinot this is performed on a block of data at a time. One of the steps in the aggregation is to copy a set of dictionary ids corresponding to the groups and to mark which groups are present in the block so they can be iterated over when aggregating the other column. The total number of groups for the entire column is known from statistics collected about the values, but not at the block level. The fused and fissured operations look like this (resetting the array would not happen for real but is necessary to make the benchmark stable):

@State(Scope.Benchmark)
public class GroupCount {

  @Param({"8", "64", "128"})
  int groups;
  @Param({"256", "512", "1024", "2048", "4096"})
  int length;

  private byte[] source;
  private byte[] dest;
  private boolean[] presence;

  @Setup(Level.Trial)
  public void setup() {
    source = new byte[length];
    dest = new byte[length];
    presence = new boolean[groups];
    SplittableRandom random = new SplittableRandom(42);
    for (int i = 0; i < source.length; i++) {
      source[i] = (byte) random.nextInt(groups);
    }
  }

  @Benchmark
  public void fused(Blackhole bh) {
    int numGroups = 0;
    for (int i = 0; i < source.length & i < dest.length; i++) {
      dest[i] = source[i];
      if (numGroups < groups && !presence[source[i]]) {
        presence[source[i] & 0xFF] = true;
        numGroups++;
      }
    }
    bh.consume(presence);
    Arrays.fill(presence, false);
  }

  @Benchmark
  public void fissured(Blackhole bh) {
    System.arraycopy(source, 0, dest, 0, source.length);
    int numGroups = 0;
    for (int i = 0; i < source.length & i < dest.length & numGroups < groups; i++) {
      if (!presence[source[i]]) {
        presence[source[i] & 0xFF] = true;
        numGroups++;
      }
    }
    bh.consume(presence);
    Arrays.fill(presence, false);
  }
}

This is a little more interesting than the other loop. Firstly, splitting the loops makes it obvious that the copy part is a manual System.arraycopy. The second loop can terminate whenever numGroups == groups without consuming the entire input, but in the fused loop has to carry on to ensure the entire (slow) manual array copy can take place. Whilst fission makes the copy fast, and an obvious candidate for removal, fission allows the second loop to terminate early.

Early termination isn’t necessarily a good thing because if the number of groups is high enough for it to be likely that no single block will contain all the groups, the loop won’t exit early. Worse, with early termination, the exit condition from the loop is data dependent, which prevents aggressive unrolling, so the loop will actually be slower when it doesn’t exit early.

Benchmark	Mode	Threads	Samples	Score	Score Error (99.9%)	Unit	Param: groups	Param: length
GroupCount.fissured	avgt	1	5	41.606830	0.187490	ns/op	8	256
GroupCount.fissured	avgt	1	5	44.518117	0.183473	ns/op	8	512
GroupCount.fissured	avgt	1	5	50.099552	0.263780	ns/op	8	1024
GroupCount.fissured	avgt	1	5	61.991607	0.254863	ns/op	8	2048
GroupCount.fissured	avgt	1	5	134.196339	0.138997	ns/op	8	4096
GroupCount.fissured	avgt	1	5	542.240261	1.993239	ns/op	64	256
GroupCount.fissured	avgt	1	5	494.577003	0.331404	ns/op	64	512
GroupCount.fissured	avgt	1	5	501.147187	0.265733	ns/op	64	1024
GroupCount.fissured	avgt	1	5	510.997369	1.264461	ns/op	64	2048
GroupCount.fissured	avgt	1	5	575.261847	0.583042	ns/op	64	4096
GroupCount.fissured	avgt	1	5	603.379813	1.389718	ns/op	128	256
GroupCount.fissured	avgt	1	5	1121.185683	26.927230	ns/op	128	512
GroupCount.fissured	avgt	1	5	1132.562254	36.868698	ns/op	128	1024
GroupCount.fissured	avgt	1	5	1141.871722	18.612546	ns/op	128	2048
GroupCount.fissured	avgt	1	5	1198.111175	49.512727	ns/op	128	4096
GroupCount.fused	avgt	1	5	197.811972	0.352569	ns/op	8	256
GroupCount.fused	avgt	1	5	344.481666	0.602220	ns/op	8	512
GroupCount.fused	avgt	1	5	642.021575	2.235174	ns/op	8	1024
GroupCount.fused	avgt	1	5	1236.629115	7.446193	ns/op	8	2048
GroupCount.fused	avgt	1	5	2478.591752	2.470746	ns/op	8	4096
GroupCount.fused	avgt	1	5	380.130086	1.642884	ns/op	64	256
GroupCount.fused	avgt	1	5	441.293203	1.776462	ns/op	64	512
GroupCount.fused	avgt	1	5	793.848668	17.267290	ns/op	64	1024
GroupCount.fused	avgt	1	5	1265.961589	5.980144	ns/op	64	2048
GroupCount.fused	avgt	1	5	2340.019651	4.249029	ns/op	64	4096
GroupCount.fused	avgt	1	5	290.768967	9.365530	ns/op	128	256
GroupCount.fused	avgt	1	5	752.002294	0.905900	ns/op	128	512
GroupCount.fused	avgt	1	5	1110.519415	1.109794	ns/op	128	1024
GroupCount.fused	avgt	1	5	1887.382144	48.908744	ns/op	128	2048
GroupCount.fused	avgt	1	5	2550.872751	23.857060	ns/op	128	4096

As can be seen above, loop fission changes things a lot, and the effect of the number of groups is more pronounced in the fissured implementation. When the ratio between length and groups is high, the fissured implementation wins (see 4096/8 134ns vs 2479ns). When the ratio is low, the effect is less extreme but in the opposite direction (see 512/128 1121ns vs 752ns). If you like looking at things like perfasm output to compare the counted and non–counted loop, it is here. Given that there are statistics about the number of groups, and the block size is a small and bounded value, fission allows a data driven decision to be made for whether to attempt to exit the second loop early.

value	$< 3$	$< 10$	$> 5$
10	0	0	1
3	0	1	0
15	0	0	1
0	1	1	0
0	1	1	0
1	1	1	0
5	0	1	0
6	0	1	1
2	1	1	0
1	1	1	0
12	0	0	1
14	0	0	1
3	0	1	0
9	0	1	1
11	0	0	1

value	bit 3	bit 2	bit 1	bit 0
10	1	0	1	0
3	0	0	1	1
15	1	1	1	1
0	0	0	0	0
0	0	0	0	0
1	0	0	0	1
5	0	1	0	1
6	0	1	1	0
2	0	0	1	0
1	0	0	0	1
12	1	1	0	0
14	1	1	1	0
3	0	0	1	1
9	1	0	0	1
11	1	0	1	1

value	bit 3	bit 2	bit 1	bit 0
10	0	1	0	1
3	1	1	0	0
15	0	0	0	0
0	1	1	1	1
0	1	1	1	1
1	1	1	1	0
5	1	0	1	0
6	1	0	0	1
2	1	1	0	1
1	1	1	1	0
12	0	0	1	1
14	0	0	0	1
3	1	1	0	0
9	0	1	1	0
11	0	1	0	0

value	$< 3$	$< 10$	$> 5$
10	0	0	1
3	0	1	0
15	0	0	1
0	1	1	0
0	1	1	0
1	1	1	0
5	0	1	0
6	0	1	1
2	1	1	0
1	1	1	0
12	0	0	1
14	0	0	1
3	0	1	0
9	0	1	1
11	0	0	1

value	bit 3	bit 2	bit 1	bit 0
10	1	0	1	0
3	0	0	1	1
15	1	1	1	1
0	0	0	0	0
0	0	0	0	0
1	0	0	0	1
5	0	1	0	1
6	0	1	1	0
2	0	0	1	0
1	0	0	0	1
12	1	1	0	0
14	1	1	1	0
3	0	0	1	1
9	1	0	0	1
11	1	0	1	1

value	bit 3	bit 2	bit 1	bit 0
10	0	1	0	1
3	1	1	0	0
15	0	0	0	0
0	1	1	1	1
0	1	1	1	1
1	1	1	1	0
5	1	0	1	0
6	1	0	0	1
2	1	1	0	1
1	1	1	1	0
12	0	0	1	1
14	0	0	0	1
3	1	1	0	0
9	0	1	1	0
11	0	1	0	0