AbstractBufWriter
Core, low-level interface
Similar to AbstractBufReader, the core interface of AbstractBufWriter consists of three functions:
get_buffer(io)returns a mutable view into the part of the buffer which is not yet used. Data is written toioby copying to the first bytes of the buffer, then callingconsume.grow_buffer(io)request to expand the buffer returned by future calls toget_buffer. This may happen by flushing data in the buffer or by reallocating a larger bufferconsume(io, n::Int)signals that the firstnbytes in the buffer are written toio, and will therefore not be returned from future calls toget_buffer.
Example: Writing v::Vector{UInt8} to an AbstractBufWriter
The method write(::AbstractBufWriter, v::Vector{UInt8}) is already implemented, but it's illustrative to see how this be implemented in terms of the core primitives above.
First, let's define it in terms of ImmutableMemoryView, and then forward the Vector method to the memory one.
using MemoryViews
# Forward the vector method to a memory view method
my_write(io::AbstractBufWriter, v::Vector{UInt8}) = my_write(io, ImmutableMemoryView(v))
function my_write(io::AbstractBufWriter, mem::ImmutableMemoryView{UInt8})::Int
n_bytes = length(mem)
while !isempty(mem)
# Get mutable buffer with uninitialized data to write to
buffer = get_buffer(io)::MutableMemoryView{UInt8}
if isempty(buffer)
# grow_buffer cannot return `nothing`, unlike for readers, but the writer
# may still be unable to add more bytes (in which case grow_buffer returns
# zero). A real implementation would use a better error
iszero(grow_buffer(io)) && error("Could not flush")
buffer = get_buffer(io)::MutableMemoryView{UInt8}
end
mn = min(length(mem), length(buffer))
# This would indicate an error in the implementation of `grow_buffer`.
# As it did not return zero, the buffer must have grown.
@assert !iszero(mn)
(to_write, mem) = split_at(mem, mn + 1)
copyto!(buffer[1:mn], to_write)
# Mark the first `mn` bytes of the buffer as being committed, thereby
# actually writing it to `io`
consume(io, mn)
end
return n_bytes
endWrapping a Base.IO
BufWriter gives any Base.IO the AbstractBufWriter interface. See the extended help of BufWriter for the precise contract.
BufferIO.get_buffer — Function
get_buffer(io::AbstractBufReader)::ImmutableMemoryView{UInt8}Get the available bytes of io.
Calling this function, even when the buffer is empty, should never do actual system I/O, and in particular should not attempt to fill the buffer. To fill the buffer, call fill_buffer.
Examples
julia> reader = BufReader(IOBuffer("abcdefghij"), 5);
julia> get_buffer(reader) |> isempty
true
julia> fill_buffer(reader)
5
julia> get_buffer(reader) |> println
UInt8[0x61, 0x62, 0x63, 0x64, 0x65]BufferIO.grow_buffer — Function
grow_buffer(io::AbstractBufWriter)::IntIncrease the amount of bytes in the writeable buffer of io if possible, returning the number of bytes added. After calling grow_buffer and getting n, the buffer obtained by get_buffer should have n more bytes.
The buffer is usually grown by flushing the buffer, expanding or reallocating the buffer. If none of these can grow the buffer, return zero.
Idiomatically, users should not call grow_buffer when the buffer is not empty, because doing so forces growing the buffer instead of letting io choose an optimal buffer size. Calling grow_buffer with a nonempty buffer is only appropriate if, for algorithmic reasons you need io buffer to be able to hold some minimum amount of data before flushing.
Examples
julia> v = VecWriter(undef, 0); get_buffer(v) |> isempty
true
julia> n_grown = grow_buffer(v); n_grown > 0
true
julia> length(get_buffer(v)) == n_grown
trueBufferIO.consume — Function
consume(io::Union{AbstractBufReader, AbstractBufWriter}, n::Int)::NothingRemove the first n bytes of the buffer of io. Consumed bytes will not be returned by future calls to get_buffer.
Calling this function must not perform I/O, but should only advance the position inside the buffer. I/O is done by fill_buffer (readers), or grow_buffer or flush (writers).
If n is negative, or larger than the current buffer size, throw an IOError with ConsumeBufferError kind. This check is a boundscheck and may be elided with @inbounds.
Examples
julia> reader = CursorReader("abcdefghij");
julia> get_buffer(reader) == b"abcdefghij"
true
julia> consume(reader, 8); get_buffer(reader) |> println
UInt8[0x69, 0x6a]
julia> consume(reader, 3) # 2 bytes remaining
ERROR: Called `consume` with a negative amount, or larger than available buffer sizeNotable AbstractWriter functions
BufferIO.get_unflushed — Function
get_unflushed(io::AbstractBufWriter)::MutableMemoryView{UInt8}Return a view into the buffered data already written to io and consumed, but not yet flushed to its underlying IO.
Bytes not appearing in the buffer may not be completely flushed if there are more layers of buffering in the IO wrapped by io. However, any bytes already consumed and not returned in get_unflushed should not be buffered in io itself.
Mutating the returned buffer is allowed, and should not cause io to malfunction. After mutating the returned buffer and calling flush, values in the updated buffer will be flushed.
This function has no default implementation and methods are optionally added to subtypes of AbstractBufWriter that can fullfil the above restrictions.
Examples
julia> io = IOBuffer(); writer = BufWriter(io);
julia> isempty(get_unflushed(writer))
true
julia> write(writer, "abc"); unflushed = get_unflushed(writer);
julia> println(unflushed)
UInt8[0x61, 0x62, 0x63]
julia> unflushed[2] = UInt8('x')
0x78
julia> flush(writer); take!(io) |> println
UInt8[0x61, 0x78, 0x63]
julia> get_unflushed(writer) |> isempty
trueBufferIO.get_nonempty_buffer — Method
get_nonempty_buffer(
io::AbstractBufWriter, min_size::Int
)::Union{Nothing, MutableMemoryView{UInt8}}Get a buffer of at least max(min_size, 1) bytes, or nothing if io cannot provide a buffer of that size. The desired buffer length is obtained by repeatedly calling grow_buffer.
Writers whose buffer can grow without bound, such as VecWriter and BufWriter, never return nothing. More generally, writers for which is_expandable returns true never return nothing. To obtain such a writer, consider using expandable.
Use of this method may cause excessive buffering without flushing, which is less memory efficient than calling the one-argument method and flushing in a loop.
Extended help
Subtypes of AbstractBufWriter may implement this method if they can provide a large buffer more efficiently than the generic fallback. The returned buffer must contain at least max(min_size, 1) bytes.
Examples
julia> using BufferIO.Testing: BoundedWriter
julia> writer = BoundedWriter(UInt8[], 4);
julia> length(get_nonempty_buffer(writer, 3)) ≥ 3
true
julia> get_nonempty_buffer(writer, 5) === nothing # buffer can hold at most 4 bytes
true
julia> length(get_nonempty_buffer(VecWriter(), 100)) ≥ 100 # VecWriter can always grow
trueWriters with a limited buffer
Some writers can't grow their buffer beyond a certain size, for example writers which compress data in fixed-size blocks. Generic code which needs a minimum amount of contiguous buffer space, e.g. to encode a value directly into the buffer, can't assume that an arbitrary AbstractBufWriter can provide it.
The trait BufferIO.is_expandable tells whether a writer type can grow its buffer to any size. For such writers, grow_buffer never returns zero, and get_nonempty_buffer(io, min_size) never returns nothing. The function expandable returns its argument if it is already expandable, and else wraps it in an ExpandWriter, which is expandable:
function write_encoded(io::AbstractBufWriter, x)
io = expandable(io)
n = encoded_size(x)
# Statically known not to be `nothing`
buffer = get_nonempty_buffer(io, n)::MutableMemoryView{UInt8}
encode!(buffer, x)
consume(io, n)
flush(io)
return n
endBufferIO.is_expandable — Function
is_expandable(T::Type{<:Union{AbstractBufReader, AbstractBufWriter}})::BoolTrait function that signals if instances of type T can grow their buffer to any size, or, for readers, until EOF.
This is a property of the type, not of the instance, and must be constant foldable. It is usually implemented as a constant true. The default is false for both readers and writers.
Readers
A reader type T must only implement is_expandable(::Type{T}) = true if fill_buffer(::T) returns an Int for every instance of T, i.e. never nothing.
In practice, this means instances of T are able to buffer their entire content with repeated calls to fill_buffer. Types for which this returns true either already buffer their whole content (such as CursorReader), or else have a growable buffer large enough to hold all the underlying data (such as BufReader or ExpandReader). For these types, get_nonempty_buffer(io, min_size) never throws an IOError of kind BufferTooShort.
Writers
A writer type T must only implement is_expandable(::Type{T}) = true if grow_buffer(::T) never returns zero for any instance of T. If such a writer can't make room for more data, e.g. because an underlying device is full, grow_buffer must throw an exception, such as an IOError of kind WriterFull. Hence, writers with a concept of being full, such as writers with a maximum size, can't be expandable.
In practice, this means instances of T can make their buffer arbitrarily large by repeated calls to grow_buffer, for example by reallocating the buffer (such as BufWriter and VecWriter), or by serving data from another, growable buffer (such as ExpandWriter).
See also: expandable
Examples
julia> BufferIO.is_expandable(BufReader)
true
julia> BufferIO.is_expandable(VecWriter)
true
julia> struct FooReader <: AbstractBufReader end
julia> BufferIO.is_expandable(FooReader) # default is false
false
julia> BufferIO.is_expandable(BufferIO.Testing.BoundedWriter)
falseBufferIO.expandable — Function
expandable(io::T)::Union{T, ExpandReader{T}} where {T <: AbstractBufReader}
expandable(io::T)::Union{T, ExpandWriter{T}} where {T <: AbstractBufWriter}Return io unchanged if BufferIO.is_expandable(typeof(io)), else wrap io in an ExpandReader or ExpandWriter and return it. The type of the result is guaranteed to have is_expandable return true, which means its buffer size is able to grow without bound, or, for readers, until EOF. Equivalently, calling fill_buffer on a resulting reader will not return nothing, and calling grow_buffer on a resulting writer will not return zero.
This function is useful when working with non-expandable IOs, or generic IOs, where the maximum buffer size is unknown, and where some minimum buffering is required. An example could be a reader that wraps a generic AbstractBufReader, and must be able to buffer a full line of input, or a writer that need to unsafe_store values of a certain size directly into the buffer of a generic AbstractBufWriter.
The concrete return type of expandable(io) is resolved at compile time.
See also: BufferIO.is_expandable
Examples
julia> io = BufReader(IOBuffer("abc"));
julia> expandable(io) === io # BufReader is already expandable
true
julia> io = BufferIO.Testing.BoundedReader("abc", 2);
julia> expandable(io) isa ExpandReader
true
julia> io = BufferIO.Testing.BoundedWriter(UInt8[], 2);
julia> expandable(io) isa ExpandWriter
true