AbstractBufWriter

Core, low-level interface

Similar to AbstractBufReader, the core interface of AbstractBufWriter consists of three functions:

  • get_buffer(io) returns a mutable view into the part of the buffer which is not yet used. Data is written to io by copying to the first bytes of the buffer, then calling consume.
  • grow_buffer(io) request to expand the buffer returned by future calls to get_buffer. This may happen by flushing data in the buffer or by reallocating a larger buffer
  • consume(io, n::Int) signals that the first n bytes in the buffer are written to io, and will therefore not be returned from future calls to get_buffer.

Example: Writing v::Vector{UInt8} to an AbstractBufWriter

The method write(::AbstractBufWriter, v::Vector{UInt8}) is already implemented, but it's illustrative to see how this be implemented in terms of the core primitives above.

First, let's define it in terms of ImmutableMemoryView, and then forward the Vector method to the memory one.

using MemoryViews

# Forward the vector method to a memory view method
my_write(io::AbstractBufWriter, v::Vector{UInt8}) = my_write(io, ImmutableMemoryView(v))

function my_write(io::AbstractBufWriter, mem::ImmutableMemoryView{UInt8})::Int
    n_bytes = length(mem)
    while !isempty(mem)
        # Get mutable buffer with uninitialized data to write to
        buffer = get_buffer(io)::MutableMemoryView{UInt8}
        if isempty(buffer)
            # grow_buffer cannot return `nothing`, unlike for readers, but the writer
            # may still be unable to add more bytes (in which case grow_buffer returns
            # zero). A real implementation would use a better error
            iszero(grow_buffer(io)) && error("Could not flush")
            buffer = get_buffer(io)::MutableMemoryView{UInt8}
        end
        mn = min(length(mem), length(buffer))
        # This would indicate an error in the implementation of `grow_buffer`.
        # As it did not return zero, the buffer must have grown.
        @assert !iszero(mn)
        (to_write, mem) = split_at(mem, mn + 1)
        copyto!(buffer[1:mn], to_write)
        # Mark the first `mn` bytes of the buffer as being committed, thereby
        # actually writing it to `io`
        consume(io, mn)
    end
    return n_bytes
end
BufferIO.get_buffer — Function
get_buffer(io::AbstractBufReader)::ImmutableMemoryView{UInt8}

Get the available bytes of io.

Calling this function, even when the buffer is empty, should never do actual system I/O, and in particular should not attempt to fill the buffer. To fill the buffer, call fill_buffer.

Examples

julia> reader = BufReader(IOBuffer("abcdefghij"), 5);

julia> get_buffer(reader) |> isempty
true

julia> fill_buffer(reader)
5

julia> get_buffer(reader) |> println
UInt8[0x61, 0x62, 0x63, 0x64, 0x65]
get_buffer(io::AbstractBufWriter)::MutableMemoryView{UInt8}

Get the available mutable buffer of io that can be written to.

Calling this function should never do actual system I/O, and in particular should not attempt to flush data from the buffer or grow the buffer. To increase the size of the buffer, call grow_buffer.

Examples

julia> writer = BufWriter(IOBuffer(), 5);

julia> buffer = get_buffer(writer);

julia> (typeof(buffer), length(buffer))
(MutableMemoryView{UInt8}, 5)

julia> write(writer, "abcde")
5

julia> get_buffer(writer) |> isempty
true

julia> flush(writer)

julia> buffer = get_buffer(writer); length(buffer)
5
BufferIO.grow_buffer — Function
grow_buffer(io::AbstractBufWriter)::Int

Increase the amount of bytes in the writeable buffer of io if possible, returning the number of bytes added. After calling grow_buffer and getting n, the buffer obtained by get_buffer should have n more bytes.

The buffer is usually grown by flushing the buffer, expanding or reallocating the buffer. If none of these can grow the buffer, return zero.

Note

Idiomatically, users should not call grow_buffer when the buffer is not empty, because doing so forces growing the buffer instead of letting io choose an optimal buffer size. Calling grow_buffer with a nonempty buffer is only appropriate if, for algorithmic reasons you need io buffer to be able to hold some minimum amount of data before flushing.

Examples

julia> v = VecWriter(undef, 0); get_buffer(v) |> isempty
true

julia> n_grown = grow_buffer(v); n_grown > 0
true

julia> length(get_buffer(v)) == n_grown
true
BufferIO.consume — Function
consume(io::Union{AbstractBufReader, AbstractBufWriter}, n::Int)::Nothing

Remove the first n bytes of the buffer of io. Consumed bytes will not be returned by future calls to get_buffer.

If n is negative, or larger than the current buffer size, throw an IOError with ConsumeBufferError kind. This check is a boundscheck and may be elided with @inbounds.

Examples

julia> reader = CursorReader("abcdefghij");

julia> get_buffer(reader) == b"abcdefghij"
true

julia> consume(reader, 8); get_buffer(reader) |> println
UInt8[0x69, 0x6a]

julia> consume(reader, 3) # 2 bytes remaining
ERROR: Called `consume` with a negative amount, or larger than available buffer size

Notable AbstractWriter functions

BufferIO.get_unflushed — Function
get_unflushed(io::AbstractBufWriter)::MutableMemoryView{UInt8}

Return a view into the buffered data already written to io and consumed, but not yet flushed to its underlying IO.

Bytes not appearing in the buffer may not be completely flushed if there are more layers of buffering in the IO wrapped by io. However, any bytes already consumed and not returned in get_unflushed should not be buffered in io itself.

Mutating the returned buffer is allowed, and should not cause io to malfunction. After mutating the returned buffer and calling flush, values in the updated buffer will be flushed.

This function has no default implementation and methods are optionally added to subtypes of AbstractBufWriter that can fullfil the above restrictions.

Examples

julia> io = IOBuffer(); writer = BufWriter(io);

julia> isempty(get_unflushed(writer))
true

julia> write(writer, "abc"); unflushed = get_unflushed(writer);

julia> println(unflushed)
UInt8[0x61, 0x62, 0x63]

julia> unflushed[2] = UInt8('x')
0x78

julia> flush(writer); take!(io) |> println
UInt8[0x61, 0x78, 0x63]

julia> get_unflushed(writer) |> isempty
true
BufferIO.get_nonempty_buffer — Method
get_nonempty_buffer(
    io::AbstractBufWriter, min_size::Int
)::Union{Nothing, MutableMemoryView{UInt8}}

Get a buffer of at least size max(min_size, 1), or nothing if that is not possible.

This method is optionally implemented for subtypes of AbstractBufWriter, and is typically only implemented for types which do not flush their data to an underlying IO, such that there is no memory savings by writing in smaller chunks.

Warning

Use of this method may cause excessive buffering without flushing, which is less memory efficient than calling the one-argument method and flushing in a loop.

Examples

julia> function write_int_le(writer::AbstractBufWriter, int::Int64)
           buf = get_nonempty_buffer(writer, sizeof(Int64))::Union{Nothing, MutableMemoryView{UInt8}}
           isnothing(buf) && throw(IOError(IOErrorKinds.BufferTooShort))
           length(buf) < sizeof(Int64) && error("Bad implementation of get_nonempty_buffer")
           GC.@preserve buf unsafe_store!(Ptr{Int64}(pointer(buf)), htol(int))
           @inbounds consume(writer, sizeof(Int64))
           return sizeof(Int64)
       end;

julia> v = VecWriter(); write_int_le(v, Int64(515))
8

julia> String(v.vec)
"\x03\x02\0\0\0\0\0\0"