Julia Snack: Duplex IOBuffers

This is a particularly niche Julia trick: IOBuffers where data is written to the end and read from the beginning.

Why are Duplex Buffers useful? Duplex Buffers facilitate arbitrary byte-wise communication between two distinct endpoints within your application. They are very akin to Julia’s built-in . In fact, for most applications one will likely prefer the latter for improved semantics and multi-processing capabilities. But sometimes, the application calls for much finer control. When the application would employ a Channel{UInt8}, this is when a developer might consider dispatching a Duplex Buffer instead, especially as such a usage pattern would likely even impede performance.

Now, Duplex Buffers are not a term used directly in the Julia documentation. “Duplex” refers to its ability to track distinct read/write pointers on the buffer — in the usage described herein: reading from the head and writing to the tail, just like (buffered) channels. Seeking out to build such behavior myself, I stumbled over a very convenient fact… They already exist.


The documentation only lists three relevant keyword arguments which may be passed to the IOBuffer constructor: read, write, and append. To achieve one half of the duplex buffer (writing to the tail end exclusively) append shall be set to true — by default, it is false. While generally, this should already suffice to implement a sort of Duplex Buffer, it comes at a rather massive disadvantage: Data which has already been read cannot be removed from the underlying buffer.

Depending on your specific use case, this newest complication may not be an issue. The buffer will be cleared when it is . But if the buffer persists in memory for an extended time, so does the “consumed” data, slowly draining the entire system of memory.

Fortunately, Julia is an open source project. The reveals a fourth flag: seekable. This flag is particularly interesting. Not only does it prevent rewinding the buffer to already read/consumed data (enforcing reading from the head exclusively), it also allows calling Base.compact on the buffer. This method will then remove all consumed data from the head of the buffer. Perfect!

The Pattern

With this knowledge, building such an IOBuffer is now child’s play:

DuplexBuffer(data::AbstractVector{UInt8} = UInt8[]) = IOBuffer(UInt8[], true, true, false, true, typemax(Int))buffer = DuplexBuffer()
write(buffer, 42)
write(buffer, 420.69)
read(buffer, Int) # == 42
write(buffer, "foobar")
read(buffer, Float64) # == 420.69
Base.compact(buffer) # now the buffer only contains "foobar"
read(buffer, String) # == "foobar"
Base.compact(buffer) # now the buffer is entirely empty

Julia’s does not directly note this usage. Hence, one should beware that this snack may break in a future version of Julia; due to semantic versioning, however, one may be fairly certain this won’t happen until version 2.0.

Of course, the example above is very uninteresting. The most likely usage involves multitasking or even multithreading, although in case of the latter the read and write operations should be (separately) synchronized across calls.

Personally, I employ this pattern in a networking application. The Duplex Buffer is used to feed arbitrary network data to specialized packet handlers, but the data is not formatted in any uniform layout.

Freelance Software Engineer | DLT/Blockchain Enthusiast & Future Entrepreneur

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store