Reference
BufferIO.AbstractBufReader — Type
abstract type AbstractBufReader endAn AbstractBufReader is a readable IO type that exposes a buffer of readable bytes to the user.
By default, subtypes of AbstractBufReader are not threadsafe, so concurrent usage should protect the instance behind a lock.
Extended help
Subtypes of this type should not have a zero-sized buffer which cannot expand when calling fill_buffer.
Subtypes T of this type should implement at least:
get_buffer(io::T)fill_buffer(io::T)consume(io::T, n::Int)Base.close(io::T)Base.isopen(io::T)
Subtypes may optionally define the following methods. See their docstring for BufReader / BufWriter for details of the implementation:
Base.seek(io::T, ::Int)relative_seek(io::T, ::Int)Base.filesize(io::T)Base.position(io::T)resize_buffer(io::T, ::Int)BufferIO.is_expandable(::Type{T})remaining_bytes_hint(io::T)unsafe_read_all!(io::T, ::Ptr{UInt8}, ::Int)get_nonempty_buffer(io::T, ::Int)
AbstractBufReaders have implementations for many Base IO methods, but with more precisely specified semantics than for Base.IO. See docstrings of the specific functions of interest.
BufferIO.AbstractBufWriter — Type
abstract type AbstractBufWriter endAn AbstractBufWriter is an IO-like type which exposes mutable memory to the user, which can be written to directly. This can help avoiding intermediate allocations when writing. For example, data can be encoded directly into the buffer of a writer, instead of first being encoded into a temporary buffer, and then copied to the writer.
By default, subtypes of AbstractBufWriter are not threadsafe, so concurrent usage should protect the instance behind a lock.
Extended help
Subtypes of this type should not have a zero-sized buffer which cannot expand when calling grow_buffer.
Subtypes T of this type should implement at least:
get_buffer(io::T)grow_buffer(io::T)consume(io::T, n::Int)Base.close(io::T)Base.isopen(io::T)Base.flush(io::T)
They may optionally implement
Base.seek(io::T, ::Int)Base.filesize(io::T)Base.position(io::T)get_unflushed(io::T)resize_buffer(io::T, ::Int)BufferIO.is_expandable(::Type{T})get_nonempty_buffer(io::T, ::Int)Base.unsafe_write(io::T, ::Ptr{UInt8}, ::UInt)
BufferIO.BufReader — Type
BufReader(f, io::IO, [buffer_size::Int])Create a BufReader wrapping io, then call f on the BufReader, and close the reader once f is finished or if it errors.
This pattern is useful for automatically cleaning up the resource of io.
julia> io = IOBuffer("hello world");
julia> BufReader(io) do reader
println(String(read(reader, 5)))
end
hello
julia> read(io) # closing reader closes io also
ERROR: ArgumentError: read failed, IOBuffer is not readable
[...]BufferIO.BufReader — Type
BufReader{T <: IO} <: AbstractBufReader
BufReader(io::IO, [buffer_size::Int])::BufReaderWrap an IO in a struct with a new buffer, giving it the AbstractBufReader interface.
The BufReader has an infinitely growable buffer, which may grow when fill_buffer is called.
When filling its buffer, the BufReader may move the buffered bytes to the beginning of its internal memory to make room for new bytes. This compaction is guaranteed to be amortized O(n_bytes_read), no matter how calls to fill_buffer and consume are interleaved.
This wrapper reads from io using readbytes!. If io is not EOF, and readbytes! reads zero bytes, an exception is thrown when filling the buffer.
Throw an ArgumentError if buffer_size is less than 1.
julia> rdr = BufReader(IOBuffer("Hello, world!\nabc\r\ndef"));
julia> get_buffer(rdr) |> isempty
true
julia> peek(rdr)
0x48
julia> readline(rdr)
"Hello, world!"
julia> String(readavailable(rdr))
"abc\r\ndef"Extended help
For an io::Base.IO type to function correctly when wrapped in a BufReader, it must implement:
eof(io)::Bool, which may block until at least one byte is available or EOF is known. If this returnsfalse, at least one byte must be available to read.- Either
readbytes!(io, v::MutableMemoryView{UInt8})::Integer, or elseunsafe_read(io, ::Ptr{UInt8}, ::UInt)andbytesavailable(io)::Integer. Ifreadbytes!are implemented, it should read at least one byte wheneof(io) === false.
Seeking, and filesize and position of the BufReader, additionally require io to implement seek, filesize and position.
BufferIO.BufWriter — Type
BufWriter(f, io::IO, [buffer_size::Int])Create a BufWriter wrapping io, then call f on the BufWriter, and close the writer once f is finished or if it errors.
This pattern is useful for automatically cleaning up the resource of io.
julia> io = IOBuffer();
julia> BufWriter(io) do writer
write(writer, "hello, world!")
shallow_flush(writer)
seekstart(io)
println(String(read(io)))
end
hello, world!
julia> iswritable(io) # closing writer closes io also
falseBufferIO.BufWriter — Type
BufWriter{T <: IO} <: AbstractBufWriter
BufWriter(io::IO, [buffer_size::Int])::BufWriterWrap an IO in a struct with a new buffer, giving it the AbstractBufWriter interface.
The BufWriter has an infinitely growable buffer, and will only expand the buffer if grow_buffer is called on it while it does not contain any data (as shown by get_unflushed).
Throw an ArgumentError if buffer_size is < 1.
julia> io = IOBuffer(); wtr = BufWriter(io);
julia> print(wtr, "Hello!")
julia> write(wtr, [0x34, 0x12, 0x78, 0x56])
4
julia> isempty(read(io)) # wtr not flushed
true
julia> flush(wtr); seekstart(io); String(read(io))
"Hello!4\x12xV"
julia> isempty(get_unflushed(wtr))
trueExtended help
For an io::Base.IO type to function correctly when wrapped in a BufWriter, it must implement:
close(io)flush(io)unsafe_write(io, ::Ptr{UInt8}, ::UInt)
Seeking, and filesize and position of the BufWriter, additionally require io to implement seek, filesize and position, where:
- Writing overwrites existing data, and seeking does not truncate
io. filesize(io)includes any data written toiobeyondposition(io). For example,IOStreammeets this, because it writes out its own buffer when seeking.
Seeking is not supported if io is in append mode (e.g. a file opened with append = true), because then io ignores its position when writing.
BufferIO.CursorReader — Type
CursorReader(x) <: AbstractBufReaderA seekable, stateful reader of the content of any object x which implements MemoryView(x)::MemoryView{UInt8}.
Closing it does nothing.
julia> rdr = CursorReader("some\ncontent\nhere");
julia> readline(rdr)
"some"
julia> read(rdr, String)
"content\nhere"
julia> seek(rdr, 8);
julia> read(rdr, String)
"tent\nhere"BufferIO.ExpandReader — Type
ExpandReader{T <: AbstractBufReader} <: AbstractBufReaderReader which wraps another reader and provides an infinitely expandable buffer. ExpandReader forwards its inner reader's buffer where possible, but if fill_buffer is called and the inner reader can't expand its buffer, ExpandReader will serve the data from its own growable buffer. This implies that fill_buffer(::ExpandReader) always returns an Int, never nothing.
ExpandReader is useful when users need to buffer a minimum amount (e.g. an entire line, or N bytes), but want to read from a generic AbstractBufReader. Generic code may use the expandable function to opt-into ExpandReader only when necessary.
ExpandReader takes control of the inner reader, and mutating the inner reader may invalidate the ExpandReader.
See also: expandable, BufferIO.is_expandable, ExpandWriter
Examples
julia> using BufferIO.Testing: BoundedReader
julia> inner = BoundedReader("Hello, world!", 4; max_fill = 4);
julia> fill_buffer(inner); fill_buffer(inner) === nothing # inner can't grow its buffer
true
julia> reader = ExpandReader(inner); # In generic code, prefer `expandable(inner)`
julia> String(get_buffer(reader)) # inner's buffer is forwarded
"Hell"
julia> while !iszero(fill_buffer(reader)) end
julia> String(get_buffer(reader)) # served from the own buffer of `reader`
"Hello, world!"
julia> BufferIO.is_expandable(typeof(reader))
trueBufferIO.ExpandWriter — Type
ExpandWriter{T <: AbstractBufWriter} <: AbstractBufWriterWriter which wraps another writer and provides an infinitely expandable buffer. ExpandWriter forwards its inner writer's buffer where possible, but if grow_buffer is called and the inner writer can't expand its buffer, ExpandWriter will serve its own growable buffer. This implies that grow_buffer(::ExpandWriter) never returns zero, and get_nonempty_buffer(::ExpandWriter, ::Int) never returns nothing. If the inner writer is full, i.e. its buffer is empty and it can't make room for more data, which may happen if e.g. the underlying device cannot store more data, these functions instead throw an IOError of kind WriterFull.
ExpandWriter is useful when users need a minimum amount of contiguous buffer space (e.g. to encode a value of N bytes directly into the buffer), but want to write to a generic AbstractBufWriter. Generic code may use the expandable function to opt-into ExpandWriter only when necessary.
Data written to the own buffer of ExpandWriter is written to the inner writer when ExpandWriter is flushed, seeked or closed, or when it needs to make room in its buffer. If the inner writer becomes full before all this data could be written to it, these operations throw an IOError of kind WriterFull.
ExpandWriter takes control of the inner writer, and mutating the inner writer may invalidate the ExpandWriter.
See also: expandable, BufferIO.is_expandable, ExpandReader
Examples
julia> using BufferIO.Testing: BoundedWriter
julia> sink = UInt8[]; inner = BoundedWriter(sink, 4);
julia> get_nonempty_buffer(inner, 8) === nothing # inner can't hold 8 bytes
true
julia> writer = ExpandWriter(inner); # In generic code, prefer `expandable(inner)`
julia> buffer = get_nonempty_buffer(writer, 8); length(buffer) ≥ 8
true
julia> buffer[1:8] .= codeunits("Hello, w"); consume(writer, 8)
julia> write(writer, "orld!")
5
julia> flush(writer); String(sink)
"Hello, world!"
julia> BufferIO.is_expandable(typeof(writer))
trueBufferIO.IOError — Type
IOErrorThis type is thrown by errors of AbstractBufReader and AbstractBufWriter. They contain the .kind::IOErrorKind public property.
See also: IOErrorKinds.IOErrorKind
Examples
julia> rdr = CursorReader("some content");
julia> try
seek(rdr, 500)
catch error
if error.kind == IOErrorKinds.BadSeek
println(stderr, "Seeking operation out of bounds")
else
rethrow()
end
end
Seeking operation out of boundsBufferIO.IOReader — Type
IOReader{T <: AbstractBufReader} <: IOWrapper type to convert an AbstractBufReader to an IO.
IOReaders implement the same part of the IO interface as AbstractBufReader, so this type is only used to satisfy type constraints.
Examples
julia> io = CursorReader("hello");
julia> f(x::IO) = String(read(x));
julia> f(io)
ERROR: MethodError: no method matching f(::CursorReader)
[...]
julia> f(IOReader(io))
"hello"BufferIO.IOWriter — Type
IOWriter{T <: AbstractBufWriter} <: IOWrapper type to convert an AbstractBufWriter to an IO.
IOWriters implement the same part of the IO interface as AbstractBufWriter, so this type is only used to satisfy type constraints.
Examples
julia> io = VecWriter();
julia> f(x::IO) = write(x, "hello");
julia> f(io)
ERROR: MethodError: no method matching f(::VecWriter)
[...]
julia> f(IOWriter(io))
5
julia> String(io.vec)
"hello"BufferIO.LineViewIterator — Type
LineViewIterator{T <: AbstractBufReader}This iterator is created by line_views(::AbstractBufReader). It is public, but is currently unexported.
It has two public properties: io.reader obtains its inner AbstractBufReader, and io.keep::Bool returns whether this iterator keeps line endings.
For semantics of iteration, see documentation of line_views(::AbstractBufReader).
BufferIO.LineWriter — Type
LineWriter{T <: AbstractBufWriter} <: IOAn io::LineWriter wraps an inner AbstractBufWriter in an IO, and flushes when a newline is written to io.
LineWriters are useful for streams like stdout, stderr or log files, where you may want automatic flushing every time a log line has been written. Newlines are defined as 0x0a bytes, even when part of binary data (e.g. a UInt32). When a single write call has multiple newlines, LineWriter may only flush at the last newline in the write, not on every newline. LineWriter may additionally flush on non-newlines, as controlled by the inner AbstractBufWriter.
LineWriter does not take control of the inner writer. The inner writer can be obtained with the public property .inner, and may be mutated by the user without this invalidating the LineWriter.
Examples
julia> (path, io) = mktemp();
julia> writer = LineWriter(BufWriter(io));
julia> write(writer, "abc\ndef\nghi")
11
julia> read(path, String)
"abc\ndef\n"
julia> close(writer.inner) # same as `close(writer)`BufferIO.VecWriter — Type
VecWriter <: AbstractBufWriterA writer that appends to a Vector{UInt8}, which is stored in the (public) property .vec.
This type is useful as an efficient string builder through takestring!(io).
Functions flush and close do not affect the writer.
Mutating io will mutate vec and vice versa. Neither vec nor io will be invalidated by mutating the other, but doing so may affect the implicit (non-semantic) behaviour (e.g. memory reallocations or efficiency) of the other. Mutating vec invalidates buffers previously obtained from io with e.g. get_buffer.
Create with one of the following constructors:
VecWriter(): Create an empty vector.VecWriter(undef, n::Int): Create an empty vector with space for at leastnbytesVecWriter(vec::Vector{UInt8}): Append tovec, which is not copied.
A VecWriter has no notion of filesize, and cannot be seeked. Instead, resize the underlying vector io.vec.
julia> vw = VecWriter();
julia> write(vw, "Hello, world!")
13
julia> append!(vw.vec, b" More data");
julia> takestring!(vw)
"Hello, world! More data"
julia> v = collect(b"abc");
julia> write(VecWriter(v), "def");
julia> String(v)
"abcdef"BufferIO.buffer_until — Method
buffer_until(
io::AbstractBufReader, byte::UInt8
)::Union{Nothing, ImmutableMemoryView{UInt8}}Get the buffer up to, and including as its last element, the first occurrence of byte, or nothing if the buffer could not grow enough to reach either EOF or an occurrence of byte.
If the initial buffer does not contain byte, fill_buffer is called until it does. On EOF, return the remainder of the buffer. This implies that calling this function on a file that's already at EOF returns an empty buffer. Therefore, this implies that either ends with byte as the only occurrence of byte in the returned buffer, or else the buffer contains all remaining data in io. These two cases can be distinguished with !isempty(buffer) && last(buffer) == byte.
If the buffer cannot be grown further, but does not contain a byte, return nothing. This is not possible for readers for which is_expandable returns true.
After calling this function, the buffer returned by get_buffer may contain more bytes than the one returned by this function. This function does not consume any bytes.
See also: get_nonempty_buffer
Examples
julia> rdr = BufReader(IOBuffer("abcdef\r\nghi"), 4);
julia> UInt8('\n') in get_buffer(rdr) # initially empty
false
julia> buffer_until(rdr, UInt8('\n')) |> String
"abcdef\r\n"BufferIO.consume — Function
consume(io::Union{AbstractBufReader, AbstractBufWriter}, n::Int)::NothingRemove the first n bytes of the buffer of io. Consumed bytes will not be returned by future calls to get_buffer.
Calling this function must not perform I/O, but should only advance the position inside the buffer. I/O is done by fill_buffer (readers), or grow_buffer or flush (writers).
If n is negative, or larger than the current buffer size, throw an IOError with ConsumeBufferError kind. This check is a boundscheck and may be elided with @inbounds.
Examples
julia> reader = CursorReader("abcdefghij");
julia> get_buffer(reader) == b"abcdefghij"
true
julia> consume(reader, 8); get_buffer(reader) |> println
UInt8[0x69, 0x6a]
julia> consume(reader, 3) # 2 bytes remaining
ERROR: Called `consume` with a negative amount, or larger than available buffer sizeBufferIO.discard_all — Method
discard_all(io::AbstractBufReader, n::Int)::IntRead n bytes from io, or until EOF, whichever comes first, and discard the read bytes. Return the number of bytes discarded.
This function is defined generically for AbstractBufReader by reading bytes, not by seeking, so it works on readers that can't seek. Subtypes of AbstractBufReader may implement this using seeking.
Throw an ArgumentError if n < 0.
See also: discard_exact, and skip to seek instead.
Examples
julia> reader = CursorReader("abcdefghij");
julia> discard_all(reader, 5)
5
julia> read(reader, 3) |> String
"fgh"
julia> discard_all(reader, 10) # 2 bytes remaining
2
julia> eof(reader)
true
julia> discard_all(reader, 100)
0BufferIO.discard_exact — Method
discard_exact(io::AbstractBufReader, n::Int)::NothingLike discard_all, but throw an IOError of kind IOErrorKinds.EOF if n bytes could not be discarded.
Examples
julia> reader = CursorReader("abcdefghij");
julia> position(reader)
0
julia> discard_exact(reader, 3)
julia> read(reader, 2) |> String
"de"
julia> discard_exact(reader, 6) # 5 bytes remaining
ERROR: End of fileBufferIO.drop_own — Method
drop_own(io::Union{ExpandReader, ExpandWriter})::Union{Nothing, Memory{UInt8}}If the own buffer of io is in use or io is closed, return nothing. Else, drop the buffer from io, and return it. The returned memory may be empty, and is owned by the caller, i.e. io is guaranteed to not keep a reference to it.
Since ExpandReaders and ExpandWriters allocate new buffers as needed, this function does not change the semantic behaviour of io, only its allocation patterns, and never puts io in an invalid state. It is intended to free memory if extra buffering is only transiently needed.
Whether this function returns nothing or the buffer depends on which buffer is used by io, which is an implementation detail.
Examples
julia> io = ExpandReader(CursorReader("Hello, world!"));
julia> String(read(io, 5))
"Hello"
julia> mem = drop_own(io);
julia> mem isa Union{Nothing, Memory{UInt8}}
true
julia> String(read(io)) # dropping does not change the content of io
", world!"BufferIO.expandable — Method
expandable(io::T)::Union{T, ExpandReader{T}} where {T <: AbstractBufReader}
expandable(io::T)::Union{T, ExpandWriter{T}} where {T <: AbstractBufWriter}Return io unchanged if BufferIO.is_expandable(typeof(io)), else wrap io in an ExpandReader or ExpandWriter and return it. The type of the result is guaranteed to have is_expandable return true, which means its buffer size is able to grow without bound, or, for readers, until EOF. Equivalently, calling fill_buffer on a resulting reader will not return nothing, and calling grow_buffer on a resulting writer will not return zero.
This function is useful when working with non-expandable IOs, or generic IOs, where the maximum buffer size is unknown, and where some minimum buffering is required. An example could be a reader that wraps a generic AbstractBufReader, and must be able to buffer a full line of input, or a writer that need to unsafe_store values of a certain size directly into the buffer of a generic AbstractBufWriter.
The concrete return type of expandable(io) is resolved at compile time.
See also: BufferIO.is_expandable
Examples
julia> io = BufReader(IOBuffer("abc"));
julia> expandable(io) === io # BufReader is already expandable
true
julia> io = BufferIO.Testing.BoundedReader("abc", 2);
julia> expandable(io) isa ExpandReader
true
julia> io = BufferIO.Testing.BoundedWriter(UInt8[], 2);
julia> expandable(io) isa ExpandWriter
trueBufferIO.fill_buffer — Function
fill_buffer(io::AbstractBufReader)::Union{Int, Nothing}Fill more bytes into the buffer from io's underlying buffer, returning the number of bytes added. After calling fill_buffer and getting n, the buffer obtained by get_buffer should have n new bytes appended.
This function may move data in the internal buffer, invalidating previous buffers obtained by get_buffer. However, data present in previous results from get_buffer must be present in the new buffer at the same indices. In other words, the following must always hold:
old_buffer = copy(get_buffer(io))
fill_buffer(io)
@assert get_buffer(io)[1:length(old_buffer)] == old_bufferThis function must fill at least one byte, except
- If the underlying io is EOF, or there is no underlying io to fill bytes from, return 0
- If the buffer is not empty, and cannot be expanded, return
nothing.
Buffered readers which do not wrap another underlying IO, and therefore can't fill its buffer should return 0 unconditionally. This function should never return nothing if the buffer is empty.
Idiomatically, users should not call fill_buffer when the buffer is not empty, because doing so may force growing the buffer instead of letting io choose an optimal buffer size. Calling fill_buffer with a nonempty buffer is only appropriate if, for algorithmic reasons you need io itself to buffer some minimum amount of data.
Examples
julia> reader = CursorReader("abcde");
julia> fill_buffer(reader) # CursorReader can't fill its buffer
0
julia> reader = BufReader(IOBuffer("abcde"), 3);
julia> length(get_buffer(reader)) # buffer of BufReader initially empty
0
julia> fill_buffer(reader)
3
julia> length(get_buffer(reader)) # now must be 0 + 3
3BufferIO.get_buffer — Function
get_buffer(io::AbstractBufReader)::ImmutableMemoryView{UInt8}Get the available bytes of io.
Calling this function, even when the buffer is empty, should never do actual system I/O, and in particular should not attempt to fill the buffer. To fill the buffer, call fill_buffer.
Examples
julia> reader = BufReader(IOBuffer("abcdefghij"), 5);
julia> get_buffer(reader) |> isempty
true
julia> fill_buffer(reader)
5
julia> get_buffer(reader) |> println
UInt8[0x61, 0x62, 0x63, 0x64, 0x65]BufferIO.get_buffer — Method
get_buffer(io::AbstractBufWriter)::MutableMemoryView{UInt8}Get the available mutable buffer of io that can be written to.
Calling this function should never do actual system I/O, and in particular should not attempt to flush data from the buffer or grow the buffer. To increase the size of the buffer, call grow_buffer.
Examples
julia> writer = BufWriter(IOBuffer(), 5);
julia> buffer = get_buffer(writer);
julia> (typeof(buffer), length(buffer))
(MutableMemoryView{UInt8}, 5)
julia> write(writer, "abcde")
5
julia> get_buffer(writer) |> isempty
true
julia> flush(writer)
julia> buffer = get_buffer(writer); length(buffer)
5BufferIO.get_nonempty_buffer — Method
get_nonempty_buffer(
io::AbstractBufReader, min_size::Int
)::Union{Nothing, ImmutableMemoryView{UInt8}}Get a buffer of at least max(min_size, 1) bytes, or nothing if io reaches EOF before buffering that amount of bytes. Throw an IOError of kind BufferTooShort if io cannot expand its buffer to max(min_size, 1) bytes. Since io can't detect EOF without room in its buffer, this may be thrown even if fewer than min_size bytes remain in io.
The desired buffer length is obtained by repeatedly calling fill_buffer.
Readers whose buffer can grow to contain the whole content of io, such as CursorReader and BufReader, never throw BufferTooShort. To obtain such a reader, consider using expandable.
Since this may grow io's internal buffer beyond min_size, users should only call this function if they need all min_size bytes to be buffered in io. Consider if you can read the data incrementally via the one-argument get_nonempty_buffer instead.
Extended help
Subtypes of AbstractBufReader may implement this method if they can provide a large buffer more efficiently than the generic fallback. The returned buffer must contain at least max(min_size, 1) bytes.
Examples
julia> using BufferIO.Testing: BoundedReader
julia> reader = BoundedReader(b"abcdefghi", 4);
julia> length(get_nonempty_buffer(reader, 3)) ≥ 3
true
julia> try
get_nonempty_buffer(reader, 5) # buffer can hold at most 4 bytes
catch error
error.kind == IOErrorKinds.BufferTooShort
end
true
julia> discard_exact(reader, 6);
julia> get_nonempty_buffer(reader, 4) === nothing # EOF after 3 bytes
trueBufferIO.get_nonempty_buffer — Method
get_nonempty_buffer(x::AbstractBufReader)::Union{Nothing, ImmutableMemoryView{UInt8}}Get a buffer with at least one byte, if bytes are available. Otherwise, fill the buffer, and return the newly filled buffer. Returns nothing only if x is EOF.
Examples
julia> reader = BufReader(IOBuffer("abc"));
julia> get_buffer(reader) |> isempty
true
julia> get_nonempty_buffer(reader) |> println
UInt8[0x61, 0x62, 0x63]
julia> consume(reader, 3)
julia> get_nonempty_buffer(reader) === nothing # EOF
trueBufferIO.get_nonempty_buffer — Method
get_nonempty_buffer(
io::AbstractBufWriter, min_size::Int
)::Union{Nothing, MutableMemoryView{UInt8}}Get a buffer of at least max(min_size, 1) bytes, or nothing if io cannot provide a buffer of that size. The desired buffer length is obtained by repeatedly calling grow_buffer.
Writers whose buffer can grow without bound, such as VecWriter and BufWriter, never return nothing. More generally, writers for which is_expandable returns true never return nothing. To obtain such a writer, consider using expandable.
Use of this method may cause excessive buffering without flushing, which is less memory efficient than calling the one-argument method and flushing in a loop.
Extended help
Subtypes of AbstractBufWriter may implement this method if they can provide a large buffer more efficiently than the generic fallback. The returned buffer must contain at least max(min_size, 1) bytes.
Examples
julia> using BufferIO.Testing: BoundedWriter
julia> writer = BoundedWriter(UInt8[], 4);
julia> length(get_nonempty_buffer(writer, 3)) ≥ 3
true
julia> get_nonempty_buffer(writer, 5) === nothing # buffer can hold at most 4 bytes
true
julia> length(get_nonempty_buffer(VecWriter(), 100)) ≥ 100 # VecWriter can always grow
trueBufferIO.get_nonempty_buffer — Method
get_nonempty_buffer(x::AbstractBufWriter)::Union{Nothing, MutableMemoryView{UInt8}}Get a buffer with at least one byte, if bytes are available. Otherwise, call grow_buffer, then get the buffer again. Returns nothing if the buffer is still empty.
Examples
julia> io = BufWriter(IOBuffer(), 12);
julia> write_repeated(io, 0x00, 12)
12
julia> get_buffer(io) |> isempty
true
julia> get_nonempty_buffer(io) |> isempty
false
julia> get_buffer(io) |> isempty # buffer now grown
falseBufferIO.get_unflushed — Method
get_unflushed(io::AbstractBufWriter)::MutableMemoryView{UInt8}Return a view into the buffered data already written to io and consumed, but not yet flushed to its underlying IO.
Bytes not appearing in the buffer may not be completely flushed if there are more layers of buffering in the IO wrapped by io. However, any bytes already consumed and not returned in get_unflushed should not be buffered in io itself.
Mutating the returned buffer is allowed, and should not cause io to malfunction. After mutating the returned buffer and calling flush, values in the updated buffer will be flushed.
This function has no default implementation and methods are optionally added to subtypes of AbstractBufWriter that can fullfil the above restrictions.
Examples
julia> io = IOBuffer(); writer = BufWriter(io);
julia> isempty(get_unflushed(writer))
true
julia> write(writer, "abc"); unflushed = get_unflushed(writer);
julia> println(unflushed)
UInt8[0x61, 0x62, 0x63]
julia> unflushed[2] = UInt8('x')
0x78
julia> flush(writer); take!(io) |> println
UInt8[0x61, 0x78, 0x63]
julia> get_unflushed(writer) |> isempty
trueBufferIO.grow_buffer — Method
grow_buffer(io::AbstractBufWriter)::IntIncrease the amount of bytes in the writeable buffer of io if possible, returning the number of bytes added. After calling grow_buffer and getting n, the buffer obtained by get_buffer should have n more bytes.
The buffer is usually grown by flushing the buffer, expanding or reallocating the buffer. If none of these can grow the buffer, return zero.
Idiomatically, users should not call grow_buffer when the buffer is not empty, because doing so forces growing the buffer instead of letting io choose an optimal buffer size. Calling grow_buffer with a nonempty buffer is only appropriate if, for algorithmic reasons you need io buffer to be able to hold some minimum amount of data before flushing.
Examples
julia> v = VecWriter(undef, 0); get_buffer(v) |> isempty
true
julia> n_grown = grow_buffer(v); n_grown > 0
true
julia> length(get_buffer(v)) == n_grown
trueBufferIO.is_expandable — Method
is_expandable(T::Type{<:Union{AbstractBufReader, AbstractBufWriter}})::BoolTrait function that signals if instances of type T can grow their buffer to any size, or, for readers, until EOF.
This is a property of the type, not of the instance, and must be constant foldable. It is usually implemented as a constant true. The default is false for both readers and writers.
Readers
A reader type T must only implement is_expandable(::Type{T}) = true if fill_buffer(::T) returns an Int for every instance of T, i.e. never nothing.
In practice, this means instances of T are able to buffer their entire content with repeated calls to fill_buffer. Types for which this returns true either already buffer their whole content (such as CursorReader), or else have a growable buffer large enough to hold all the underlying data (such as BufReader or ExpandReader). For these types, get_nonempty_buffer(io, min_size) never throws an IOError of kind BufferTooShort.
Writers
A writer type T must only implement is_expandable(::Type{T}) = true if grow_buffer(::T) never returns zero for any instance of T. If such a writer can't make room for more data, e.g. because an underlying device is full, grow_buffer must throw an exception, such as an IOError of kind WriterFull. Hence, writers with a concept of being full, such as writers with a maximum size, can't be expandable.
In practice, this means instances of T can make their buffer arbitrarily large by repeated calls to grow_buffer, for example by reallocating the buffer (such as BufWriter and VecWriter), or by serving data from another, growable buffer (such as ExpandWriter).
See also: expandable
Examples
julia> BufferIO.is_expandable(BufReader)
true
julia> BufferIO.is_expandable(VecWriter)
true
julia> struct FooReader <: AbstractBufReader end
julia> BufferIO.is_expandable(FooReader) # default is false
false
julia> BufferIO.is_expandable(BufferIO.Testing.BoundedWriter)
falseBufferIO.line_views — Method
line_views(x::AbstractBufReader; keep::Bool=false)::LineViewIterator{typeof(x)}Create an iterator of lines of x. The returned views are ImmutableMemoryView{UInt8} into x's buffer. Use StringView to turn the lines into AbstractStrings.
The views may be invalidated when mutating x, which may happen on subsequent iterations of the iterator. See extended help for more precise semantics.
If keep is false (the default), the final trailing \r\n or \n bytes are removed from each line.
Examples
julia> lines = line_views(CursorReader("abc\r\ndef\n\r\ngh"));
julia> first(lines) |> String
"abc"
julia> first(lines) == b"abc"
true
julia> first(iterate(lines, 9)) |> String
""
julia> sum(length, lines)
2
julia> isempty(lines)
trueExtended help
A line is defined as all data up to and including \n (0x0a) or \r\n (0x0d 0x0a), or the remainder of the data in io if no \n byte was found. If the input is empty, this iterator is also empty.
If x has a limited buffer size, and cannot grow its buffer, and an entire line cannot be kept in the buffer, an IOError is thrown, with its kind being IOErrorKinds.BufferTooShort. To iterate lines of any length from such a reader, use line_views(expandable(x)).
The resulting iterator will NOT close x when exhausted, this must be handled elsewhere.
Iterator state and io advancement
The resulting iterator itr::LineViewIterator's state is guaranteed, public interface:
iterate(itr)is equivalent toiterate(itr, 0)iterate(itr, n::Int)is equivalent toconsume(itr.reader, n); iterate(itr)ifn> 0- The state returned by
iterateis anIntequal to the length of the line emitted, plus the number of stripped\r\nor\nbytes, unlesskeep.
These semantics together mean that a normal for-loop will exhaust the underlying io, and that no emitted line will be invalidated before the next call to iterate.
As an example, to read two lines, one may do:
# Read first line, and do something with it
(line_1, s) = iterate(itr); x = process(line_1)
# Two argument iterate advances itr to the begining of next line.
(line_2, s) = iterate(itr, s); y = process(line_2)
# Advance iterator from second line to third line
consume(itr.reader, s) # .reader is a public propertyBufferIO.line_views — Method
line_views(x::MemoryView{UInt8}; keep::Bool=false)Return a stateless iterator of the lines in x. The returned views are ImmutableMemoryView{UInt8} views into x. Use StringView to turn them into AbstractStrings.
A line is defined as all data up to and including \n (0x0a) or \r\n (0x0d 0x0a), or the remainder of the data in io if no \n byte was found. If the input is empty, this iterator is also empty.
If keep is false (the default), any trailing \r\n or \n is removed from each line.
Examples
julia> mem = MemoryView("abc\r\ndef\nab\n");
julia> foreach(i -> println(repr(String(i))), line_views(mem))
"abc"
"def"
"ab"BufferIO.read_all! — Method
read_all!(io::AbstractBufReader, dst::MutableMemoryView{UInt8})::IntRead bytes into dst until either dst is filled or io is EOF, returning the number of bytes read.
Examples
julia> reader = BufReader(IOBuffer("abcdefgh"), 3);
julia> v = zeros(UInt8, 10);
julia> read_all!(reader, MemoryView(v))
8
julia> String(v)
"abcdefgh\0\0"BufferIO.read_into! — Method
read_into!(x::AbstractBufReader, dst::MutableMemoryView{UInt8})::IntRead bytes into the beginning of dst, returning the number of bytes read. This function will always read at least 1 byte, except when dst is empty, or x is EOF.
This function is defined generically for AbstractBufReader. New methods should strive to do at most one read call to the underlying IO, if x wraps such an IO.
Examples
julia> reader = CursorReader("abcde");
julia> v = zeros(UInt8, 8);
julia> read_into!(reader, MemoryView(v))
5
julia> println(v)
UInt8[0x61, 0x62, 0x63, 0x64, 0x65, 0x00, 0x00, 0x00]BufferIO.relative_seek — Method
relative_seek(io::AbstractBufReader, delta::Int) -> ioSeek the stream by delta relative to the current position. Seeking by a positive delta will advance the stream, and a negative delta rewinds the stream. When delta is zero, the stream is unchanged.
Seeking beyond either end of the stream will throw an IOError of kind BadSeek.
This method can be faster than seek, because it does not require io to query its absolute position, and therefore, in certain circumstances, the seeking can occur purely in-memory without interacting with the underlying IO.
BufferIO.remaining_bytes_hint — Method
remaining_bytes_hint(io::Union{AbstractBufReader, IO})::Union{Nothing, Int}Return an estimate of the number of bytes remaining in io before EOF, including the bytes buffered in io, or nothing if no estimate is available. A hint of zero means io is estimated to be at EOF, not that the size is unknown. The hint must not be negative. If io implements filesize and position, the hint should usually be filesize(io) - position(io).
This is only a hint, which callers can use to e.g. allocate memory up front. The actual number of remaining bytes may be smaller or larger than the hint, for example if the underlying file is modified concurrently, or if the operating system reports an inaccurate file size. Hence, callers must remain correct when the hint is wrong.
Calling this function must not change the state of io. In particular, it must not fill the buffer of io, or read from it. It should be cheap compared to reading from io, but it may query metadata, such as the size of an underlying file, with a system call.
Extended help
The default implementation returns nothing.
Methods are defined for the readers in BufferIO, and for the some Base.IO types. Users may add it to their own Base.IO subtypes to allow BufReaders wrapping their type to make use of the hint.
Examples
julia> reader = CursorReader("abcdef");
julia> remaining_bytes_hint(reader)
6
julia> discard_exact(reader, 6); remaining_bytes_hint(reader)
0
julia> reader = BufReader(IOBuffer("abc")); fill_buffer(reader);
julia> remaining_bytes_hint(reader) # includes buffered bytes
3
julia> remaining_bytes_hint(BufferIO.Testing.BoundedReader("abc", 2)) === nothing # unknown
trueBufferIO.resize_buffer — Method
resize_buffer(io::Union{BufWriter, BufReader}, n::Int) -> ioResize the internal buffer of io to exactly n bytes.
Throw an ArgumentError if n is less than 1, or lower than the currently number of buffered bytes (length of get_unflushed for BufWriter, length of get_buffer for BufReader).
julia> w = BufWriter(IOBuffer());
julia> write(w, "abc")
3
julia> length(get_buffer(resize_buffer(w, 5)))
2
julia> resize_buffer(w, 2)
ERROR: ArgumentError: Buffer size smaller than current number of buffered bytes
[...]
julia> shallow_flush(w)
3
julia> resize_buffer(w, 2) === w
trueBufferIO.shallow_flush — Method
shallow_flush(io::BufWriter)::IntClear the buffer of io by writing to the wrapped IO, but do not flush the wrapped IO. Return the number of bytes flushed.
This function is only defined for BufWriter, not generically for AbstractBufWriter. Generic code should use flush instead.
julia> io = IOBuffer();
julia> wtr = BufWriter(io);
julia> write(wtr, "hello!");
julia> take!(io)
UInt8[]
julia> shallow_flush(wtr)
6
julia> String(take!(io))
"hello!"BufferIO.take_inner — Function
take_inner(io)Close io without closing the inner io wrapped by io, and return the inner io.
If io is a writer, any data buffered in io is first written to the inner io, so no written data is lost. The inner io is not flushed.
As with close, io should not be used after calling take_inner, and isopen(io) returns false, unless io remains usable. In particular, operations on io may mutate the wrapped IO, and so may invalidate it. Calling take_inner on an io which is closed, or whose wrapped IO has already been taken, is not allowed.
Implementations may, but are not required to, detect and prevent using io after take_inner, or calling take_inner on a closed io, by throwing an IOError of kind ClosedIO.
Examples
julia> cursor = CursorReader("abc");
julia> reader = IOReader(cursor);
julia> take_inner(reader) === cursor
trueBufferIO.take_inner — Method
take_inner(io::BufReader{T})::TClose io and return its wrapped IO, without closing the wrapped IO.
Any data buffered in io and not yet consumed is discarded. To keep it, copy it from get_buffer(io) before calling take_inner.
After this, all operations on io which would access the wrapped IO throw an IOError of kind ClosedIO, and close(io) does nothing.
Throw an IOError of kind ClosedIO if io is already closed.
Examples
julia> reader = BufReader(IOBuffer("abcdef"), 4);
julia> String(read(reader, 2))
"ab"
julia> io = take_inner(reader); # "cd" was buffered in reader, and is discarded
julia> String(read(io))
"ef"
julia> read(reader, UInt8)
ERROR: Unsupported operation on closed IO
[...]BufferIO.take_inner — Method
take_inner(io::BufWriter{T})::TWrite the buffered data of io to its wrapped IO using shallow_flush, then close io and return the wrapped IO. The wrapped IO is neither closed not flushed.
After this, all operations on io which would access the wrapped IO throw an IOError of kind ClosedIO, and close(io) does nothing.
Throw an IOError of kind ClosedIO if io is already closed.
Examples
julia> buf = IOBuffer(); writer = BufWriter(buf);
julia> write(writer, "abc");
julia> take_inner(writer) === buf
true
julia> String(take!(buf)) # buffered data was written
"abc"
julia> flush(writer)
ERROR: Unsupported operation on closed IO
[...]BufferIO.take_inner — Method
take_inner(io::ExpandReader{T})::TClose io and return its inner reader, without closing the inner reader.
Any data buffered in io is consumed, and the returned inner reader is positioned at the position of io after the consumption. To keep the data that would be consumed by take_inner, copy it from get_buffer(io) before calling take_inner.
After this, all operations on io which would access the inner reader throw an IOError of kind ClosedIO, and close(io) does nothing.
Throw an IOError of kind ClosedIO if io is already closed.
Examples
julia> reader = ExpandReader(BufReader(IOBuffer("abcdef"), 4));
julia> String(read(reader, 2))
"ab"
julia> kept = String(get_buffer(reader))
"cd"
julia> inner = take_inner(reader); # "cd" is consumed
julia> String(read(inner)) # inner continues right after `kept`
"ef"
julia> read(reader, UInt8)
ERROR: Unsupported operation on closed IO
[...]BufferIO.take_inner — Method
take_inner(io::ExpandWriter{T})::TWrite the data buffered in the own buffer of io to its inner writer, then close io and return the inner writer. The inner writer is neither closed nor flushed.
After this, all operations on io which would access the inner writer throw an IOError of kind ClosedIO, and close(io) does nothing.
Throw an IOError of kind ClosedIO if io is already closed, or of kind WriterFull if the inner writer can't hold the data buffered in io. In the latter case, io is not closed.
Examples
julia> using BufferIO.Testing: BoundedWriter
julia> sink = UInt8[]; inner = BoundedWriter(sink, 4);
julia> writer = ExpandWriter(inner);
julia> write_repeated(writer, UInt8('a'), 3); fill!(get_nonempty_buffer(writer, 6), UInt8('b'));
julia> consume(writer, 6);
julia> take_inner(writer) === inner
true
julia> flush(inner); String(sink)
"aaabbbbbb"
julia> write(writer, 0x01)
ERROR: Unsupported operation on closed IO
[...]BufferIO.unsafe_read_all! — Method
unsafe_read_all!(io::AbstractBufReader, p::Ptr{UInt8}, nbytes::Int)::IntCopy bytes from io to the memory pointed to by p until either nbytes bytes are copied or io is EOF, returning the number of bytes copied. This is the unsafe counterpart of read_all!.
Throw an ArgumentError if nbytes is negative.
See also: unsafe_read, which throws if io is EOF before nbytes bytes are copied.
Safety: The caller is responsible for ensuring that p is valid, GC protected, and points to at least nbytes bytes of writeable memory.
Examples
julia> reader = CursorReader("abcdef");
julia> v = zeros(UInt8, 8);
julia> GC.@preserve v unsafe_read_all!(reader, pointer(v), 2)
2
julia> GC.@preserve v unsafe_read_all!(reader, pointer(v, 3), 6) # 4 bytes remaining
4
julia> String(v[1:6])
"abcdef"BufferIO.write_repeated — Method
write_repeated(io::AbstractBufWriter, byte::UInt8, n::Int)::IntWrite byte to io n times, and return n. This is equivalent to write(io, fill(byte, n)), but more efficient.
Throw an InexactError if n is negative. If io cannot make room for all n bytes, throw an IOError of kind IOErrorKinds.WriterFull. In that case, some bytes may already have been written.
Examples
julia> w = VecWriter(collect(b"abc"));
julia> write_repeated(w, UInt8('x'), 6)
6
julia> takestring!(w)
"abcxxxxxx"BufferIO.IOErrorKinds.IOErrorKind — Type
IOErrorKindEnum indicating what error was thrown. The current list is non-exhaustive, and more may be added in future releases. The integral value of these enums are subject to change in minor versions.
Current errors:
BufferIO.IOErrorKinds — Module
module IOErrorKindsUsed as a namespace for IOErrorKind.
BufferIO.IOErrorKinds.ConsumeBufferError — Constant
ConsumeBufferErrorOccurs when calling consume with a negative amount of bytes, or with more bytes than length(get_buffer(io)).
BufferIO.IOErrorKinds.BadSeek — Constant
BadSeekAn out-of-bounds seek operation was attempted.
BufferIO.IOErrorKinds.EOF — Constant
EOFOccurs when trying a reading operation on a file that has reached end-of-file.
BufferIO.IOErrorKinds.BufferTooShort — Constant
BufferTooShortThrown by various functions that require a minimum buffer size, which the io cannot provide. This should only be thrown if the buffer is unable to grow to the required size, and not if e.g. the buffer does not expand because the io is EOF.
BufferIO.IOErrorKinds.WriterFull — Constant
WriterFullOccurs when a writer cannot make room for more data, i.e. when grow_buffer returns zero, such that a write operation cannot complete. If a writer can make room, but not a contiguous buffer of a required minimum size, BufferTooShort is thrown instead.
BufferIO.IOErrorKinds.ClosedIO — Constant
ClosedIOIndicates an operation was done on a closed IO.
Base.readavailable — Method
readavailable(io::AbstractBufReader)::Vector{UInt8}Read the available bytes of io to a new Vector{UInt8}, except if zero bytes are available. In that case, it will attempt to get more bytes exactly once. If still no bytes are available, io is EOF, and the resulting vector is empty.
Examples
julia> io = BufReader(IOBuffer("hello, world!"), 10);
julia> bytesavailable(io)
0
julia> readavailable(io) == b"hello, wor"
true
julia> readavailable(io) == b"ld!"
true
julia> readavailable(io) == b""
trueBase.peek — Method
peek(io::AbstractBufReader, UInt8)::UInt8Get the next UInt8 in io, without advancing the IO, or throw an IOError containing IOErrorKinds.EOF if io is EOF.
Examples
julia> io = CursorReader("x");
julia> peek(io, UInt8) === UInt8('x')
true
julia> read(io, UInt8) == UInt8('x') # now EOF
true
julia> peek(io, UInt8)
ERROR: End of file
[...]Base.read — Method
read(io::AbstractBufReader, UInt8)::UInt8Get the next UInt8 in io, or throw an IOError containing IOErrorKinds.EOF if io is EOF.
Examples
julia> io = CursorReader("xy");
julia> read(io, UInt8) === UInt8('x')
true
julia> read(io, UInt8) === UInt8('y')
true
julia> read(io, UInt8)
ERROR: End of file
[...]Base.readbytes! — Function
readbytes!(io::AbstractBufReader, b::AbstractVector{UInt8}, nb::Integer=length(b))::IntRead at most nb bytes from io into b, returning the number of bytes read. If nb > 0, this function will read zero bytes if and only if io is EOF.
b must use one-based indexing. The size of b will be increased if needed (i.e. if nb is greater than length(b) and enough bytes could be read), but it will never be decreased.
It is generally preferred to use read_into! instead of this method.
Base.unsafe_read — Method
unsafe_read(io::AbstractBufReader, p::Ptr{UInt8}, nbytes::Integer)::NothingCopy exactly nbytes bytes from io to the memory pointed to by p. If io reaches EOF before nbytes bytes are copied, throw an IOError of kind IOErrorKinds.EOF. In that case, the bytes before EOF may already have been copied.
Throw an InexactError if nbytes is negative.
See also: unsafe_read_all!, which stops at EOF instead of throwing.
Safety: The caller is responsible for ensuring that p is valid, GC protected, and points to at least nbytes bytes of writeable memory.
Examples
julia> reader = CursorReader("abcdef");
julia> v = zeros(UInt8, 4);
julia> GC.@preserve v unsafe_read(reader, pointer(v), 4)
julia> String(v)
"abcd"
julia> GC.@preserve v unsafe_read(reader, pointer(v), 4) # 2 bytes remaining
ERROR: End of file
[...]Base.copyline — Method
copyline(
out::Union{IO, AbstractBufWriter},
from::AbstractBufReader;
keep::Bool = false
) -> outCopy one line from from to out, returning out. A line is defined as data up to and including \n (byte 0x0a), or all remaining data in from if no such byte is present.
If keep is false, as it is by default, the trailing \r\n (encoded as 0x0d 0x0a) or \n will not be copied to out, but it will be consumed from from.
Base.close — Method
close(io::Union{AbstractBufReader, AbstractBufWriter})Close io, releasing its underlying resources, such as a wrapped IO or a file handle. If io is a writer, close first flushes io, so no written data is lost.
Closing is idempotent: Closing an io which is already closed does nothing.
After io is closed, the behaviour of operations on io is unspecified, except that close and isopen behave as documented, and that no operation causes undefined behaviour. Depending on the implementation, operations on a closed io may throw an IOError of kind ClosedIO, or continue to work normally.
See also: take_inner
Examples
julia> reader = BufReader(IOBuffer("abc"));
julia> close(reader); close(reader) # closing twice does nothing
julia> isopen(reader)
false
julia> read(reader)
ERROR: Unsupported operation on closed IO
[...]Base.isopen — Method
isopen(io::Union{AbstractBufReader, AbstractBufWriter})::BoolReturn whether io is still usable, that is, return false if and only if io has been made unusable by close or take_inner, or a similar function.
For types that remain usable after close, such as VecWriter, isopen should return true, even after close has been called on the io. Hence, isopen(io) does not generically tell whether close(io) has been called.
Examples
julia> reader = BufReader(IOBuffer("abc"));
julia> isopen(reader)
true
julia> close(reader); isopen(reader)
false
julia> cursor = CursorReader("abc");
julia> close(cursor); isopen(cursor) # closing a CursorReader does nothing
trueBase.filesize — Method
filesize(io::AbstractBufReader)::IntGet the total size, in bytes, which can be read by io, and the span in which io can be seeked. Types implementing filesize should also implement seek.
The filesize does not depend on the current reading state of the io, i.e. reading bytes should not change the filesize.
Base.filesize — Method
filesize(io::AbstractBufWriter)::IntGet the size, in bytes, of the data written to io, including data which has been committed with consume, but not yet flushed. This is the size the underlying file or IO will have once io is flushed, provided nothing else modifies it.
0:filesize(io) is the range of valid offsets for seek, and position(io) is always in this range. Writing beyond the end of the data increases the filesize. Other operations, including seeking and flushing, do not change it.
Types implementing filesize should also implement seek and position.
Examples
julia> io = BufWriter(IOBuffer());
julia> write(io, "abc");
julia> filesize(io) # includes unflushed data
3
julia> seek(io, 1); filesize(io)
3
julia> write(io, "xyz"); filesize(io)
4Base.seek — Method
seek(io::AbstractBufReader, offset::Integer) -> ioSeek io to the zero-based position offset, if io supports seeking, and return io. When offset === 0, this is equivalent to seekstart. If filesize(io) is implemented, seek(io, filesize(io)) is equivalent to seekend(io).
Valid offsets are 0:filesize(io), if io implements filesize. Seeking outside these bounds throws an IOError of kind BadSeek.
This method is not generically defined for AbstractBufReader.
julia> rdr = BufReader(IOBuffer("Hello, world!"));
julia> String(read(rdr, 5))
"Hello"
julia> seek(rdr, 3);
julia> String(read(rdr, 5))
"lo, w"
julia> seek(rdr, 13);
julia> read(rdr)
UInt8[]Base.seek — Method
seek(io::AbstractBufWriter, offset::Integer) -> ioSeek io to the zero-based position offset, and return io. Subsequent writes write (or overwrite) data starting at offset.
Valid values for offset are in 0:filesize(io), if filesize(io) is defined. Seeking outside these bounds throws an IOError of kind BadSeek, and leaves io unchanged.
Seeking never changes the filesize, nor any data already written to io, including data not yet flushed. In particular, seeking never truncates the underlying file or IO.
Seeking may write buffered data to the underlying IO, but need not flush it. Use flush for that.
This method is not generically defined for AbstractBufWriter. Implementors of seek should also define filesize(io) and position(io)
Examples
julia> buf = IOBuffer();
julia> io = BufWriter(buf);
julia> write(io, "abcdef");
julia> seek(io, 3);
julia> write(io, "xyzghi");
julia> seek(io, 10)
ERROR: Invalid seek, possibly seek out of bounds
[...]
julia> flush(io);
julia> String(take!(buf))
"abcxyzghi"Base.skip — Method
skip(io::AbstractBufReader, offset::Integer) -> ioSeek io by offset relative to its current position, and return io.
Like relative_seek, this requires io to support seeking, and seeking beyond either end of io throws an IOError of kind BadSeek. To advance a reader that may not support seeking, by reading and discarding bytes, use discard_all or discard_exact.
Examples
julia> reader = CursorReader("abcdef");
julia> skip(reader, 4); String(read(reader, 1))
"e"
julia> skip(reader, -3); String(read(reader))
"cdef"
julia> skip(reader, 1)
ERROR: Invalid seek, possibly seek out of bounds
[...]Base.position — Method
Base.position(io::AbstractBufWriter)::IntGet the zero-based stream position.
If the stream position is p (zero-based), then the next byte written will be byte number p + 1 (one-based) in the file. The stream position accounts for buffered (consumed, but unflushed) bytes. The value of position is always in 0:filesize(io), if filesize is defined.
Examples
julia> io = BufWriter(IOBuffer());
julia> write(io, "hello");
julia> (filesize(io), position(io))
(5, 5)
julia> seek(io, 2);
julia> (filesize(io), position(io))
(5, 2)Base.position — Method
Base.position(io::AbstractBufReader)::IntGet the zero-based stream position.
If the stream position is p (zero-based), then the next byte read will be byte number p + 1 (one-based). The value of position must be in 0:filesize(io), if filesize is defined.
Examples
julia> reader = CursorReader("abcdefghij");
julia> position(reader) # zero-indexed
0
julia> read(reader, 3) |> String
"abc"
julia> position(reader)
3
julia> read(reader, 2) |> String
"de"
julia> position(reader)
5
julia> seekstart(reader); position(reader)
0Testing utilities
BufferIO.Testing — Module
module BufferIO.TestingMinimal implementations of AbstractBufReader and AbstractBufWriter intended for testing code that is generic over buffered readers and writers, similar in spirit to Test.GenericString.
The types in this module implement only the required interface methods, and have deliberately restricted buffers, or buffers which are filled in controlled increments. This exercises the code paths that BufReader or VecWriter never hit, such as fill_buffer returning nothing, and fallback definitions of optional methods. Furthermore, the readers overwrite their old buffer whenever views of their buffer may be invalidated, which helps to catch code that uses such views after they have been invalidated.
These types are not optimized for performance, and should only be used in tests.
See also: BoundedReader, ChunkedReader, BoundedWriter
BufferIO.Testing.BoundedReader — Type
BoundedReader(data, max_buffer_size::Int; max_fill::Int = 1) <: AbstractBufReaderReader of the bytes of data, whose buffer can hold at most max_buffer_size bytes, and where each call to fill_buffer adds at most max_fill bytes. data can be any object x which implements ImmutableMemoryView{UInt8}(x).
When the buffer holds max_buffer_size bytes, fill_buffer returns nothing, and so BufferIO.is_expandable(BoundedReader) is false. This makes BoundedReader useful to test that generic code correctly handles readers with a limited buffer size, and, with a small max_fill, readers which fill their buffer in small increments.
Views returned by get_buffer are invalidated by fill_buffer, consume and close. To catch code which nonetheless uses such views, every call to these functions moves the buffer to new memory, and overwrites the old memory by flipping all bits of every byte. Hence, invalidated views will appear to contain different data. The bytes of data itself are never modified.
BoundedReader only implements the methods required of an AbstractBufReader, so optional methods such as seek and position are unsupported or use the generic fallback definitions.
Examples
julia> using BufferIO.Testing: BoundedReader
julia> reader = BoundedReader("abcdefg", 4; max_fill = 3);
julia> fill_buffer(reader), fill_buffer(reader), fill_buffer(reader)
(3, 1, nothing)
julia> buffer = get_buffer(reader); String(buffer)
"abcd"
julia> consume(reader, 3); buffer == b"abcd" # buffer invalidated by consume
false
julia> String(read(reader))
"defg"
julia> BufferIO.is_expandable(BoundedReader)
falseBufferIO.Testing.ChunkedReader — Type
ChunkedReader(chunks) <: AbstractBufReaderReader of the bytes of chunks, where the nth call to fill_buffer adds exactly the bytes of the nth chunk to the buffer, and returns 0 once all chunks have been added. Each chunk can be any object x which implements ImmutableMemoryView{UInt8}(x), and must not be empty, since fill_buffer returning 0 signals EOF.
The buffer can grow without limit, so fill_buffer never returns nothing, and BufferIO.is_expandable(ChunkedReader) is true. This makes ChunkedReader useful to test that generic code correctly handles data which arrives in pieces, by controlling exactly where the buffer ends after each call to fill_buffer, e.g. in the middle of a line, or between a \r and a \n.
Like BoundedReader, every call to fill_buffer, consume and close moves the buffer to new memory, and overwrites the old memory by flipping all bits of every byte, to catch code which uses views of the buffer after they have been invalidated. The bytes of the chunks themselves are never modified.
ChunkedReader only implements the methods required of an AbstractBufReader, so optional methods such as seek and position are unsupported or use the generic fallback definitions.
Examples
julia> using BufferIO.Testing: ChunkedReader
julia> reader = ChunkedReader(["abc\r", "\nde", "f"]);
julia> fill_buffer(reader), fill_buffer(reader)
(4, 3)
julia> String(get_buffer(reader))
"abc\r\nde"
julia> consume(reader, 5); fill_buffer(reader), fill_buffer(reader)
(1, 0)
julia> String(read(reader))
"def"
julia> BufferIO.is_expandable(ChunkedReader)
trueBufferIO.Testing.BoundedWriter — Type
BoundedWriter(sink::Vector{UInt8}, max_buffer_size::Int) <: AbstractBufWriterWriter with a fixed-size buffer of max_buffer_size bytes, which is never expanded. When flushed, or when grow_buffer is called, all buffered bytes are appended to sink.
This makes BoundedWriter useful to test that generic code correctly handles writers with a limited buffer size.
BoundedWriter only implements the methods required of an AbstractBufWriter, so optional methods such as seek and position are unsupported or use the generic fallback definitions. BufferIO.is_expandable(BoundedWriter) is false.
Examples
julia> using BufferIO.Testing: BoundedWriter
julia> sink = UInt8[]; writer = BoundedWriter(sink, 4);
julia> length(get_buffer(writer))
4
julia> write(writer, "Hello, world!")
13
julia> String(copy(sink)) # 12 bytes flushed, 1 still buffered
"Hello, world"
julia> flush(writer); String(sink)
"Hello, world!"