Reference

BufferIO.AbstractBufReader — Type
abstract type AbstractBufReader end

An AbstractBufReader is a readable IO type that exposes a buffer of readable bytes to the user.

Warning

By default, subtypes of AbstractBufReader are not threadsafe, so concurrent usage should protect the instance behind a lock.

Extended help

Subtypes of this type should not have a zero-sized buffer which cannot expand when calling fill_buffer.

Subtypes T of this type should implement at least:

  • get_buffer(io::T)
  • fill_buffer(io::T)
  • consume(io::T, n::Int)
  • Base.close(io::T)
  • Base.isopen(io::T)

Subtypes may optionally define the following methods. See their docstring for BufReader / BufWriter for details of the implementation:

  • Base.seek(io::T, ::Int)
  • relative_seek(io::T, ::Int)
  • Base.filesize(io::T)
  • Base.position(io::T)
  • resize_buffer(io::T, ::Int)
  • BufferIO.is_expandable(::Type{T})
  • remaining_bytes_hint(io::T)
  • unsafe_read_all!(io::T, ::Ptr{UInt8}, ::Int)
  • get_nonempty_buffer(io::T, ::Int)

AbstractBufReaders have implementations for many Base IO methods, but with more precisely specified semantics than for Base.IO. See docstrings of the specific functions of interest.

BufferIO.AbstractBufWriter — Type
abstract type AbstractBufWriter end

An AbstractBufWriter is an IO-like type which exposes mutable memory to the user, which can be written to directly. This can help avoiding intermediate allocations when writing. For example, data can be encoded directly into the buffer of a writer, instead of first being encoded into a temporary buffer, and then copied to the writer.

Warning

By default, subtypes of AbstractBufWriter are not threadsafe, so concurrent usage should protect the instance behind a lock.

Extended help

Subtypes of this type should not have a zero-sized buffer which cannot expand when calling grow_buffer.

Subtypes T of this type should implement at least:

  • get_buffer(io::T)
  • grow_buffer(io::T)
  • consume(io::T, n::Int)
  • Base.close(io::T)
  • Base.isopen(io::T)
  • Base.flush(io::T)

They may optionally implement

  • Base.seek(io::T, ::Int)
  • Base.filesize(io::T)
  • Base.position(io::T)
  • get_unflushed(io::T)
  • resize_buffer(io::T, ::Int)
  • BufferIO.is_expandable(::Type{T})
  • get_nonempty_buffer(io::T, ::Int)
  • Base.unsafe_write(io::T, ::Ptr{UInt8}, ::UInt)
BufferIO.BufReader — Type
BufReader(f, io::IO, [buffer_size::Int])

Create a BufReader wrapping io, then call f on the BufReader, and close the reader once f is finished or if it errors.

This pattern is useful for automatically cleaning up the resource of io.

julia> io = IOBuffer("hello world");

julia> BufReader(io) do reader
           println(String(read(reader, 5)))
       end
hello

julia> read(io) # closing reader closes io also
ERROR: ArgumentError: read failed, IOBuffer is not readable
[...]
BufferIO.BufReader — Type
BufReader{T <: IO} <: AbstractBufReader
BufReader(io::IO, [buffer_size::Int])::BufReader

Wrap an IO in a struct with a new buffer, giving it the AbstractBufReader interface.

The BufReader has an infinitely growable buffer, which may grow when fill_buffer is called.

When filling its buffer, the BufReader may move the buffered bytes to the beginning of its internal memory to make room for new bytes. This compaction is guaranteed to be amortized O(n_bytes_read), no matter how calls to fill_buffer and consume are interleaved.

This wrapper reads from io using readbytes!. If io is not EOF, and readbytes! reads zero bytes, an exception is thrown when filling the buffer.

Throw an ArgumentError if buffer_size is less than 1.

julia> rdr = BufReader(IOBuffer("Hello, world!\nabc\r\ndef"));

julia> get_buffer(rdr) |> isempty
true

julia> peek(rdr)
0x48

julia> readline(rdr)
"Hello, world!"

julia> String(readavailable(rdr))
"abc\r\ndef"

Extended help

For an io::Base.IO type to function correctly when wrapped in a BufReader, it must implement:

  • eof(io)::Bool, which may block until at least one byte is available or EOF is known. If this returns false, at least one byte must be available to read.
  • Either readbytes!(io, v::MutableMemoryView{UInt8})::Integer, or else unsafe_read(io, ::Ptr{UInt8}, ::UInt) and bytesavailable(io)::Integer. If readbytes! are implemented, it should read at least one byte when eof(io) === false.

Seeking, and filesize and position of the BufReader, additionally require io to implement seek, filesize and position.

BufferIO.BufWriter — Type
BufWriter(f, io::IO, [buffer_size::Int])

Create a BufWriter wrapping io, then call f on the BufWriter, and close the writer once f is finished or if it errors.

This pattern is useful for automatically cleaning up the resource of io.

julia> io = IOBuffer();

julia> BufWriter(io) do writer
           write(writer, "hello, world!")
           shallow_flush(writer)
           seekstart(io)
           println(String(read(io)))
       end
hello, world!

julia> iswritable(io) # closing writer closes io also
false
BufferIO.BufWriter — Type
BufWriter{T <: IO} <: AbstractBufWriter
BufWriter(io::IO, [buffer_size::Int])::BufWriter

Wrap an IO in a struct with a new buffer, giving it the AbstractBufWriter interface.

The BufWriter has an infinitely growable buffer, and will only expand the buffer if grow_buffer is called on it while it does not contain any data (as shown by get_unflushed).

Throw an ArgumentError if buffer_size is < 1.

julia> io = IOBuffer(); wtr = BufWriter(io);

julia> print(wtr, "Hello!")

julia> write(wtr, [0x34, 0x12, 0x78, 0x56])
4

julia> isempty(read(io)) # wtr not flushed
true

julia> flush(wtr); seekstart(io); String(read(io))
"Hello!4\x12xV"

julia> isempty(get_unflushed(wtr))
true

Extended help

For an io::Base.IO type to function correctly when wrapped in a BufWriter, it must implement:

  • close(io)
  • flush(io)
  • unsafe_write(io, ::Ptr{UInt8}, ::UInt)

Seeking, and filesize and position of the BufWriter, additionally require io to implement seek, filesize and position, where:

  • Writing overwrites existing data, and seeking does not truncate io.
  • filesize(io) includes any data written to io beyond position(io). For example, IOStream meets this, because it writes out its own buffer when seeking.

Seeking is not supported if io is in append mode (e.g. a file opened with append = true), because then io ignores its position when writing.

BufferIO.CursorReader — Type
CursorReader(x) <: AbstractBufReader

A seekable, stateful reader of the content of any object x which implements MemoryView(x)::MemoryView{UInt8}.

Closing it does nothing.

julia> rdr = CursorReader("some\ncontent\nhere");

julia> readline(rdr)
"some"

julia> read(rdr, String)
"content\nhere"

julia> seek(rdr, 8);

julia> read(rdr, String)
"tent\nhere"
BufferIO.ExpandReader — Type
ExpandReader{T <: AbstractBufReader} <: AbstractBufReader

Reader which wraps another reader and provides an infinitely expandable buffer. ExpandReader forwards its inner reader's buffer where possible, but if fill_buffer is called and the inner reader can't expand its buffer, ExpandReader will serve the data from its own growable buffer. This implies that fill_buffer(::ExpandReader) always returns an Int, never nothing.

ExpandReader is useful when users need to buffer a minimum amount (e.g. an entire line, or N bytes), but want to read from a generic AbstractBufReader. Generic code may use the expandable function to opt-into ExpandReader only when necessary.

ExpandReader takes control of the inner reader, and mutating the inner reader may invalidate the ExpandReader.

See also: expandable, BufferIO.is_expandable, ExpandWriter

Examples

julia> using BufferIO.Testing: BoundedReader

julia> inner = BoundedReader("Hello, world!", 4; max_fill = 4);

julia> fill_buffer(inner); fill_buffer(inner) === nothing # inner can't grow its buffer
true

julia> reader = ExpandReader(inner); # In generic code, prefer `expandable(inner)`

julia> String(get_buffer(reader)) # inner's buffer is forwarded
"Hell"

julia> while !iszero(fill_buffer(reader)) end

julia> String(get_buffer(reader)) # served from the own buffer of `reader`
"Hello, world!"

julia> BufferIO.is_expandable(typeof(reader))
true
BufferIO.ExpandWriter — Type
ExpandWriter{T <: AbstractBufWriter} <: AbstractBufWriter

Writer which wraps another writer and provides an infinitely expandable buffer. ExpandWriter forwards its inner writer's buffer where possible, but if grow_buffer is called and the inner writer can't expand its buffer, ExpandWriter will serve its own growable buffer. This implies that grow_buffer(::ExpandWriter) never returns zero, and get_nonempty_buffer(::ExpandWriter, ::Int) never returns nothing. If the inner writer is full, i.e. its buffer is empty and it can't make room for more data, which may happen if e.g. the underlying device cannot store more data, these functions instead throw an IOError of kind WriterFull.

ExpandWriter is useful when users need a minimum amount of contiguous buffer space (e.g. to encode a value of N bytes directly into the buffer), but want to write to a generic AbstractBufWriter. Generic code may use the expandable function to opt-into ExpandWriter only when necessary.

Data written to the own buffer of ExpandWriter is written to the inner writer when ExpandWriter is flushed, seeked or closed, or when it needs to make room in its buffer. If the inner writer becomes full before all this data could be written to it, these operations throw an IOError of kind WriterFull.

ExpandWriter takes control of the inner writer, and mutating the inner writer may invalidate the ExpandWriter.

See also: expandable, BufferIO.is_expandable, ExpandReader

Examples

julia> using BufferIO.Testing: BoundedWriter

julia> sink = UInt8[]; inner = BoundedWriter(sink, 4);

julia> get_nonempty_buffer(inner, 8) === nothing # inner can't hold 8 bytes
true

julia> writer = ExpandWriter(inner); # In generic code, prefer `expandable(inner)`

julia> buffer = get_nonempty_buffer(writer, 8); length(buffer) ≥ 8
true

julia> buffer[1:8] .= codeunits("Hello, w"); consume(writer, 8)

julia> write(writer, "orld!")
5

julia> flush(writer); String(sink)
"Hello, world!"

julia> BufferIO.is_expandable(typeof(writer))
true
BufferIO.IOError — Type
IOError

This type is thrown by errors of AbstractBufReader and AbstractBufWriter. They contain the .kind::IOErrorKind public property.

See also: IOErrorKinds.IOErrorKind

Examples

julia> rdr = CursorReader("some content");

julia> try
           seek(rdr, 500)
       catch error
           if error.kind == IOErrorKinds.BadSeek
               println(stderr, "Seeking operation out of bounds")
           else
               rethrow()
           end
        end
Seeking operation out of bounds
BufferIO.IOReader — Type
IOReader{T <: AbstractBufReader} <: IO

Wrapper type to convert an AbstractBufReader to an IO.

IOReaders implement the same part of the IO interface as AbstractBufReader, so this type is only used to satisfy type constraints.

Examples

julia> io = CursorReader("hello");

julia> f(x::IO) = String(read(x));

julia> f(io)
ERROR: MethodError: no method matching f(::CursorReader)
[...]

julia> f(IOReader(io))
"hello"
BufferIO.IOWriter — Type
IOWriter{T <: AbstractBufWriter} <: IO

Wrapper type to convert an AbstractBufWriter to an IO.

IOWriters implement the same part of the IO interface as AbstractBufWriter, so this type is only used to satisfy type constraints.

Examples

julia> io = VecWriter();

julia> f(x::IO) = write(x, "hello");

julia> f(io)
ERROR: MethodError: no method matching f(::VecWriter)
[...]

julia> f(IOWriter(io))
5

julia> String(io.vec)
"hello"
BufferIO.LineWriter — Type
LineWriter{T <: AbstractBufWriter} <: IO

An io::LineWriter wraps an inner AbstractBufWriter in an IO, and flushes when a newline is written to io.

LineWriters are useful for streams like stdout, stderr or log files, where you may want automatic flushing every time a log line has been written. Newlines are defined as 0x0a bytes, even when part of binary data (e.g. a UInt32). When a single write call has multiple newlines, LineWriter may only flush at the last newline in the write, not on every newline. LineWriter may additionally flush on non-newlines, as controlled by the inner AbstractBufWriter.

LineWriter does not take control of the inner writer. The inner writer can be obtained with the public property .inner, and may be mutated by the user without this invalidating the LineWriter.

Examples

julia> (path, io) = mktemp();

julia> writer = LineWriter(BufWriter(io));

julia> write(writer, "abc\ndef\nghi")
11

julia> read(path, String)
"abc\ndef\n"

julia> close(writer.inner) # same as `close(writer)`
BufferIO.VecWriter — Type
VecWriter <: AbstractBufWriter

A writer that appends to a Vector{UInt8}, which is stored in the (public) property .vec.

This type is useful as an efficient string builder through takestring!(io).

Functions flush and close do not affect the writer.

Mutating io will mutate vec and vice versa. Neither vec nor io will be invalidated by mutating the other, but doing so may affect the implicit (non-semantic) behaviour (e.g. memory reallocations or efficiency) of the other. Mutating vec invalidates buffers previously obtained from io with e.g. get_buffer.

Create with one of the following constructors:

  • VecWriter(): Create an empty vector.
  • VecWriter(undef, n::Int): Create an empty vector with space for at least n bytes
  • VecWriter(vec::Vector{UInt8}): Append to vec, which is not copied.

A VecWriter has no notion of filesize, and cannot be seeked. Instead, resize the underlying vector io.vec.

julia> vw = VecWriter();

julia> write(vw, "Hello, world!")
13

julia> append!(vw.vec, b" More data");

julia> takestring!(vw)
"Hello, world! More data"

julia> v = collect(b"abc");

julia> write(VecWriter(v), "def");

julia> String(v)
"abcdef"
BufferIO.buffer_until — Method
buffer_until(
    io::AbstractBufReader, byte::UInt8
)::Union{Nothing, ImmutableMemoryView{UInt8}}

Get the buffer up to, and including as its last element, the first occurrence of byte, or nothing if the buffer could not grow enough to reach either EOF or an occurrence of byte.

If the initial buffer does not contain byte, fill_buffer is called until it does. On EOF, return the remainder of the buffer. This implies that calling this function on a file that's already at EOF returns an empty buffer. Therefore, this implies that either ends with byte as the only occurrence of byte in the returned buffer, or else the buffer contains all remaining data in io. These two cases can be distinguished with !isempty(buffer) && last(buffer) == byte.

If the buffer cannot be grown further, but does not contain a byte, return nothing. This is not possible for readers for which is_expandable returns true.

After calling this function, the buffer returned by get_buffer may contain more bytes than the one returned by this function. This function does not consume any bytes.

See also: get_nonempty_buffer

Examples

julia> rdr = BufReader(IOBuffer("abcdef\r\nghi"), 4);

julia> UInt8('\n') in get_buffer(rdr) # initially empty
false

julia> buffer_until(rdr, UInt8('\n')) |> String
"abcdef\r\n"
BufferIO.consume — Function
consume(io::Union{AbstractBufReader, AbstractBufWriter}, n::Int)::Nothing

Remove the first n bytes of the buffer of io. Consumed bytes will not be returned by future calls to get_buffer.

Calling this function must not perform I/O, but should only advance the position inside the buffer. I/O is done by fill_buffer (readers), or grow_buffer or flush (writers).

If n is negative, or larger than the current buffer size, throw an IOError with ConsumeBufferError kind. This check is a boundscheck and may be elided with @inbounds.

Examples

julia> reader = CursorReader("abcdefghij");

julia> get_buffer(reader) == b"abcdefghij"
true

julia> consume(reader, 8); get_buffer(reader) |> println
UInt8[0x69, 0x6a]

julia> consume(reader, 3) # 2 bytes remaining
ERROR: Called `consume` with a negative amount, or larger than available buffer size
BufferIO.discard_all — Method
discard_all(io::AbstractBufReader, n::Int)::Int

Read n bytes from io, or until EOF, whichever comes first, and discard the read bytes. Return the number of bytes discarded.

This function is defined generically for AbstractBufReader by reading bytes, not by seeking, so it works on readers that can't seek. Subtypes of AbstractBufReader may implement this using seeking.

Throw an ArgumentError if n < 0.

See also: discard_exact, and skip to seek instead.

Examples

julia> reader = CursorReader("abcdefghij");

julia> discard_all(reader, 5)
5

julia> read(reader, 3) |> String
"fgh"

julia> discard_all(reader, 10) # 2 bytes remaining
2

julia> eof(reader)
true

julia> discard_all(reader, 100)
0
BufferIO.discard_exact — Method
discard_exact(io::AbstractBufReader, n::Int)::Nothing

Like discard_all, but throw an IOError of kind IOErrorKinds.EOF if n bytes could not be discarded.

Examples

julia> reader = CursorReader("abcdefghij");

julia> position(reader)
0

julia> discard_exact(reader, 3)

julia> read(reader, 2) |> String
"de"

julia> discard_exact(reader, 6) # 5 bytes remaining
ERROR: End of file
BufferIO.drop_own — Method
drop_own(io::Union{ExpandReader, ExpandWriter})::Union{Nothing, Memory{UInt8}}

If the own buffer of io is in use or io is closed, return nothing. Else, drop the buffer from io, and return it. The returned memory may be empty, and is owned by the caller, i.e. io is guaranteed to not keep a reference to it.

Since ExpandReaders and ExpandWriters allocate new buffers as needed, this function does not change the semantic behaviour of io, only its allocation patterns, and never puts io in an invalid state. It is intended to free memory if extra buffering is only transiently needed.

Whether this function returns nothing or the buffer depends on which buffer is used by io, which is an implementation detail.

Examples

julia> io = ExpandReader(CursorReader("Hello, world!"));

julia> String(read(io, 5))
"Hello"

julia> mem = drop_own(io);

julia> mem isa Union{Nothing, Memory{UInt8}}
true

julia> String(read(io)) # dropping does not change the content of io
", world!"
BufferIO.expandable — Method
expandable(io::T)::Union{T, ExpandReader{T}} where {T <: AbstractBufReader}
expandable(io::T)::Union{T, ExpandWriter{T}} where {T <: AbstractBufWriter}

Return io unchanged if BufferIO.is_expandable(typeof(io)), else wrap io in an ExpandReader or ExpandWriter and return it. The type of the result is guaranteed to have is_expandable return true, which means its buffer size is able to grow without bound, or, for readers, until EOF. Equivalently, calling fill_buffer on a resulting reader will not return nothing, and calling grow_buffer on a resulting writer will not return zero.

This function is useful when working with non-expandable IOs, or generic IOs, where the maximum buffer size is unknown, and where some minimum buffering is required. An example could be a reader that wraps a generic AbstractBufReader, and must be able to buffer a full line of input, or a writer that need to unsafe_store values of a certain size directly into the buffer of a generic AbstractBufWriter.

The concrete return type of expandable(io) is resolved at compile time.

See also: BufferIO.is_expandable

Examples

julia> io = BufReader(IOBuffer("abc"));

julia> expandable(io) === io # BufReader is already expandable
true

julia> io = BufferIO.Testing.BoundedReader("abc", 2);

julia> expandable(io) isa ExpandReader
true

julia> io = BufferIO.Testing.BoundedWriter(UInt8[], 2);

julia> expandable(io) isa ExpandWriter
true
BufferIO.fill_buffer — Function
fill_buffer(io::AbstractBufReader)::Union{Int, Nothing}

Fill more bytes into the buffer from io's underlying buffer, returning the number of bytes added. After calling fill_buffer and getting n, the buffer obtained by get_buffer should have n new bytes appended.

This function may move data in the internal buffer, invalidating previous buffers obtained by get_buffer. However, data present in previous results from get_buffer must be present in the new buffer at the same indices. In other words, the following must always hold:

old_buffer = copy(get_buffer(io))
fill_buffer(io)
@assert get_buffer(io)[1:length(old_buffer)] == old_buffer

This function must fill at least one byte, except

  • If the underlying io is EOF, or there is no underlying io to fill bytes from, return 0
  • If the buffer is not empty, and cannot be expanded, return nothing.

Buffered readers which do not wrap another underlying IO, and therefore can't fill its buffer should return 0 unconditionally. This function should never return nothing if the buffer is empty.

Note

Idiomatically, users should not call fill_buffer when the buffer is not empty, because doing so may force growing the buffer instead of letting io choose an optimal buffer size. Calling fill_buffer with a nonempty buffer is only appropriate if, for algorithmic reasons you need io itself to buffer some minimum amount of data.

Examples

julia> reader = CursorReader("abcde");

julia> fill_buffer(reader) # CursorReader can't fill its buffer
0

julia> reader = BufReader(IOBuffer("abcde"), 3);

julia> length(get_buffer(reader)) # buffer of BufReader initially empty
0

julia> fill_buffer(reader)
3

julia> length(get_buffer(reader)) # now must be 0 + 3
3
BufferIO.get_buffer — Function
get_buffer(io::AbstractBufReader)::ImmutableMemoryView{UInt8}

Get the available bytes of io.

Calling this function, even when the buffer is empty, should never do actual system I/O, and in particular should not attempt to fill the buffer. To fill the buffer, call fill_buffer.

Examples

julia> reader = BufReader(IOBuffer("abcdefghij"), 5);

julia> get_buffer(reader) |> isempty
true

julia> fill_buffer(reader)
5

julia> get_buffer(reader) |> println
UInt8[0x61, 0x62, 0x63, 0x64, 0x65]
BufferIO.get_buffer — Method
get_buffer(io::AbstractBufWriter)::MutableMemoryView{UInt8}

Get the available mutable buffer of io that can be written to.

Calling this function should never do actual system I/O, and in particular should not attempt to flush data from the buffer or grow the buffer. To increase the size of the buffer, call grow_buffer.

Examples

julia> writer = BufWriter(IOBuffer(), 5);

julia> buffer = get_buffer(writer);

julia> (typeof(buffer), length(buffer))
(MutableMemoryView{UInt8}, 5)

julia> write(writer, "abcde")
5

julia> get_buffer(writer) |> isempty
true

julia> flush(writer)

julia> buffer = get_buffer(writer); length(buffer)
5
BufferIO.get_nonempty_buffer — Method
get_nonempty_buffer(
    io::AbstractBufReader, min_size::Int
)::Union{Nothing, ImmutableMemoryView{UInt8}}

Get a buffer of at least max(min_size, 1) bytes, or nothing if io reaches EOF before buffering that amount of bytes. Throw an IOError of kind BufferTooShort if io cannot expand its buffer to max(min_size, 1) bytes. Since io can't detect EOF without room in its buffer, this may be thrown even if fewer than min_size bytes remain in io.

The desired buffer length is obtained by repeatedly calling fill_buffer.

Readers whose buffer can grow to contain the whole content of io, such as CursorReader and BufReader, never throw BufferTooShort. To obtain such a reader, consider using expandable.

Warning

Since this may grow io's internal buffer beyond min_size, users should only call this function if they need all min_size bytes to be buffered in io. Consider if you can read the data incrementally via the one-argument get_nonempty_buffer instead.

Extended help

Subtypes of AbstractBufReader may implement this method if they can provide a large buffer more efficiently than the generic fallback. The returned buffer must contain at least max(min_size, 1) bytes.

Examples

julia> using BufferIO.Testing: BoundedReader

julia> reader = BoundedReader(b"abcdefghi", 4);

julia> length(get_nonempty_buffer(reader, 3)) ≥ 3
true

julia> try
           get_nonempty_buffer(reader, 5) # buffer can hold at most 4 bytes
       catch error
           error.kind == IOErrorKinds.BufferTooShort
       end
true

julia> discard_exact(reader, 6);

julia> get_nonempty_buffer(reader, 4) === nothing # EOF after 3 bytes
true
BufferIO.get_nonempty_buffer — Method
get_nonempty_buffer(x::AbstractBufReader)::Union{Nothing, ImmutableMemoryView{UInt8}}

Get a buffer with at least one byte, if bytes are available. Otherwise, fill the buffer, and return the newly filled buffer. Returns nothing only if x is EOF.

Examples

julia> reader = BufReader(IOBuffer("abc"));

julia> get_buffer(reader) |> isempty
true

julia> get_nonempty_buffer(reader) |> println
UInt8[0x61, 0x62, 0x63]

julia> consume(reader, 3)

julia> get_nonempty_buffer(reader) === nothing # EOF
true
BufferIO.get_nonempty_buffer — Method
get_nonempty_buffer(
    io::AbstractBufWriter, min_size::Int
)::Union{Nothing, MutableMemoryView{UInt8}}

Get a buffer of at least max(min_size, 1) bytes, or nothing if io cannot provide a buffer of that size. The desired buffer length is obtained by repeatedly calling grow_buffer.

Writers whose buffer can grow without bound, such as VecWriter and BufWriter, never return nothing. More generally, writers for which is_expandable returns true never return nothing. To obtain such a writer, consider using expandable.

Warning

Use of this method may cause excessive buffering without flushing, which is less memory efficient than calling the one-argument method and flushing in a loop.

Extended help

Subtypes of AbstractBufWriter may implement this method if they can provide a large buffer more efficiently than the generic fallback. The returned buffer must contain at least max(min_size, 1) bytes.

Examples

julia> using BufferIO.Testing: BoundedWriter

julia> writer = BoundedWriter(UInt8[], 4);

julia> length(get_nonempty_buffer(writer, 3)) ≥ 3
true

julia> get_nonempty_buffer(writer, 5) === nothing # buffer can hold at most 4 bytes
true

julia> length(get_nonempty_buffer(VecWriter(), 100)) ≥ 100 # VecWriter can always grow
true
BufferIO.get_nonempty_buffer — Method
get_nonempty_buffer(x::AbstractBufWriter)::Union{Nothing, MutableMemoryView{UInt8}}

Get a buffer with at least one byte, if bytes are available. Otherwise, call grow_buffer, then get the buffer again. Returns nothing if the buffer is still empty.

Examples

julia> io = BufWriter(IOBuffer(), 12);

julia> write_repeated(io, 0x00, 12)
12

julia> get_buffer(io) |> isempty
true

julia> get_nonempty_buffer(io) |> isempty
false

julia> get_buffer(io) |> isempty # buffer now grown
false
BufferIO.get_unflushed — Method
get_unflushed(io::AbstractBufWriter)::MutableMemoryView{UInt8}

Return a view into the buffered data already written to io and consumed, but not yet flushed to its underlying IO.

Bytes not appearing in the buffer may not be completely flushed if there are more layers of buffering in the IO wrapped by io. However, any bytes already consumed and not returned in get_unflushed should not be buffered in io itself.

Mutating the returned buffer is allowed, and should not cause io to malfunction. After mutating the returned buffer and calling flush, values in the updated buffer will be flushed.

This function has no default implementation and methods are optionally added to subtypes of AbstractBufWriter that can fullfil the above restrictions.

Examples

julia> io = IOBuffer(); writer = BufWriter(io);

julia> isempty(get_unflushed(writer))
true

julia> write(writer, "abc"); unflushed = get_unflushed(writer);

julia> println(unflushed)
UInt8[0x61, 0x62, 0x63]

julia> unflushed[2] = UInt8('x')
0x78

julia> flush(writer); take!(io) |> println
UInt8[0x61, 0x78, 0x63]

julia> get_unflushed(writer) |> isempty
true
BufferIO.grow_buffer — Method
grow_buffer(io::AbstractBufWriter)::Int

Increase the amount of bytes in the writeable buffer of io if possible, returning the number of bytes added. After calling grow_buffer and getting n, the buffer obtained by get_buffer should have n more bytes.

The buffer is usually grown by flushing the buffer, expanding or reallocating the buffer. If none of these can grow the buffer, return zero.

Note

Idiomatically, users should not call grow_buffer when the buffer is not empty, because doing so forces growing the buffer instead of letting io choose an optimal buffer size. Calling grow_buffer with a nonempty buffer is only appropriate if, for algorithmic reasons you need io buffer to be able to hold some minimum amount of data before flushing.

Examples

julia> v = VecWriter(undef, 0); get_buffer(v) |> isempty
true

julia> n_grown = grow_buffer(v); n_grown > 0
true

julia> length(get_buffer(v)) == n_grown
true
BufferIO.is_expandable — Method
is_expandable(T::Type{<:Union{AbstractBufReader, AbstractBufWriter}})::Bool

Trait function that signals if instances of type T can grow their buffer to any size, or, for readers, until EOF.

This is a property of the type, not of the instance, and must be constant foldable. It is usually implemented as a constant true. The default is false for both readers and writers.

Readers

A reader type T must only implement is_expandable(::Type{T}) = true if fill_buffer(::T) returns an Int for every instance of T, i.e. never nothing.

In practice, this means instances of T are able to buffer their entire content with repeated calls to fill_buffer. Types for which this returns true either already buffer their whole content (such as CursorReader), or else have a growable buffer large enough to hold all the underlying data (such as BufReader or ExpandReader). For these types, get_nonempty_buffer(io, min_size) never throws an IOError of kind BufferTooShort.

Writers

A writer type T must only implement is_expandable(::Type{T}) = true if grow_buffer(::T) never returns zero for any instance of T. If such a writer can't make room for more data, e.g. because an underlying device is full, grow_buffer must throw an exception, such as an IOError of kind WriterFull. Hence, writers with a concept of being full, such as writers with a maximum size, can't be expandable.

In practice, this means instances of T can make their buffer arbitrarily large by repeated calls to grow_buffer, for example by reallocating the buffer (such as BufWriter and VecWriter), or by serving data from another, growable buffer (such as ExpandWriter).

See also: expandable

Examples

julia> BufferIO.is_expandable(BufReader)
true

julia> BufferIO.is_expandable(VecWriter)
true

julia> struct FooReader <: AbstractBufReader end

julia> BufferIO.is_expandable(FooReader) # default is false
false

julia> BufferIO.is_expandable(BufferIO.Testing.BoundedWriter)
false
BufferIO.line_views — Method
line_views(x::AbstractBufReader; keep::Bool=false)::LineViewIterator{typeof(x)}

Create an iterator of lines of x. The returned views are ImmutableMemoryView{UInt8} into x's buffer. Use StringView to turn the lines into AbstractStrings.

The views may be invalidated when mutating x, which may happen on subsequent iterations of the iterator. See extended help for more precise semantics.

If keep is false (the default), the final trailing \r\n or \n bytes are removed from each line.

Examples

julia> lines = line_views(CursorReader("abc\r\ndef\n\r\ngh"));

julia> first(lines) |> String
"abc"

julia> first(lines) == b"abc"
true

julia> first(iterate(lines, 9)) |> String
""

julia> sum(length, lines)
2

julia> isempty(lines)
true

Extended help

A line is defined as all data up to and including \n (0x0a) or \r\n (0x0d 0x0a), or the remainder of the data in io if no \n byte was found. If the input is empty, this iterator is also empty.

If x has a limited buffer size, and cannot grow its buffer, and an entire line cannot be kept in the buffer, an IOError is thrown, with its kind being IOErrorKinds.BufferTooShort. To iterate lines of any length from such a reader, use line_views(expandable(x)).

The resulting iterator will NOT close x when exhausted, this must be handled elsewhere.

Iterator state and io advancement

The resulting iterator itr::LineViewIterator's state is guaranteed, public interface:

  • iterate(itr) is equivalent to iterate(itr, 0)
  • iterate(itr, n::Int) is equivalent to consume(itr.reader, n); iterate(itr) if n > 0
  • The state returned by iterate is an Int equal to the length of the line emitted, plus the number of stripped \r\n or \n bytes, unless keep.

These semantics together mean that a normal for-loop will exhaust the underlying io, and that no emitted line will be invalidated before the next call to iterate.

As an example, to read two lines, one may do:

# Read first line, and do something  with it
(line_1, s) = iterate(itr);    x = process(line_1)
# Two argument iterate advances itr to the begining of next line.
(line_2, s) = iterate(itr, s); y = process(line_2)
# Advance iterator from second line to third line
consume(itr.reader, s) # .reader is a public property
BufferIO.line_views — Method
line_views(x::MemoryView{UInt8}; keep::Bool=false)

Return a stateless iterator of the lines in x. The returned views are ImmutableMemoryView{UInt8} views into x. Use StringView to turn them into AbstractStrings.

A line is defined as all data up to and including \n (0x0a) or \r\n (0x0d 0x0a), or the remainder of the data in io if no \n byte was found. If the input is empty, this iterator is also empty.

If keep is false (the default), any trailing \r\n or \n is removed from each line.

Examples

julia> mem = MemoryView("abc\r\ndef\nab\n");

julia> foreach(i -> println(repr(String(i))), line_views(mem))
"abc"
"def"
"ab"
BufferIO.read_all! — Method
read_all!(io::AbstractBufReader, dst::MutableMemoryView{UInt8})::Int

Read bytes into dst until either dst is filled or io is EOF, returning the number of bytes read.

Examples

julia> reader = BufReader(IOBuffer("abcdefgh"), 3);

julia> v = zeros(UInt8, 10);

julia> read_all!(reader, MemoryView(v))
8

julia> String(v)
"abcdefgh\0\0"
BufferIO.read_into! — Method
read_into!(x::AbstractBufReader, dst::MutableMemoryView{UInt8})::Int

Read bytes into the beginning of dst, returning the number of bytes read. This function will always read at least 1 byte, except when dst is empty, or x is EOF.

This function is defined generically for AbstractBufReader. New methods should strive to do at most one read call to the underlying IO, if x wraps such an IO.

Examples

julia> reader = CursorReader("abcde");

julia> v = zeros(UInt8, 8);

julia> read_into!(reader, MemoryView(v))
5

julia> println(v)
UInt8[0x61, 0x62, 0x63, 0x64, 0x65, 0x00, 0x00, 0x00]
BufferIO.relative_seek — Method
relative_seek(io::AbstractBufReader, delta::Int) -> io

Seek the stream by delta relative to the current position. Seeking by a positive delta will advance the stream, and a negative delta rewinds the stream. When delta is zero, the stream is unchanged.

Seeking beyond either end of the stream will throw an IOError of kind BadSeek.

This method can be faster than seek, because it does not require io to query its absolute position, and therefore, in certain circumstances, the seeking can occur purely in-memory without interacting with the underlying IO.

BufferIO.remaining_bytes_hint — Method
remaining_bytes_hint(io::Union{AbstractBufReader, IO})::Union{Nothing, Int}

Return an estimate of the number of bytes remaining in io before EOF, including the bytes buffered in io, or nothing if no estimate is available. A hint of zero means io is estimated to be at EOF, not that the size is unknown. The hint must not be negative. If io implements filesize and position, the hint should usually be filesize(io) - position(io).

This is only a hint, which callers can use to e.g. allocate memory up front. The actual number of remaining bytes may be smaller or larger than the hint, for example if the underlying file is modified concurrently, or if the operating system reports an inaccurate file size. Hence, callers must remain correct when the hint is wrong.

Calling this function must not change the state of io. In particular, it must not fill the buffer of io, or read from it. It should be cheap compared to reading from io, but it may query metadata, such as the size of an underlying file, with a system call.

Extended help

The default implementation returns nothing.

Methods are defined for the readers in BufferIO, and for the some Base.IO types. Users may add it to their own Base.IO subtypes to allow BufReaders wrapping their type to make use of the hint.

Examples

julia> reader = CursorReader("abcdef");

julia> remaining_bytes_hint(reader)
6

julia> discard_exact(reader, 6); remaining_bytes_hint(reader)
0

julia> reader = BufReader(IOBuffer("abc")); fill_buffer(reader);

julia> remaining_bytes_hint(reader) # includes buffered bytes
3

julia> remaining_bytes_hint(BufferIO.Testing.BoundedReader("abc", 2)) === nothing # unknown
true
BufferIO.resize_buffer — Method
resize_buffer(io::Union{BufWriter, BufReader}, n::Int) -> io

Resize the internal buffer of io to exactly n bytes.

Throw an ArgumentError if n is less than 1, or lower than the currently number of buffered bytes (length of get_unflushed for BufWriter, length of get_buffer for BufReader).

julia> w = BufWriter(IOBuffer());

julia> write(w, "abc")
3

julia> length(get_buffer(resize_buffer(w, 5)))
2

julia> resize_buffer(w, 2)
ERROR: ArgumentError: Buffer size smaller than current number of buffered bytes
[...]

julia> shallow_flush(w)
3

julia> resize_buffer(w, 2) === w
true
BufferIO.shallow_flush — Method
shallow_flush(io::BufWriter)::Int

Clear the buffer of io by writing to the wrapped IO, but do not flush the wrapped IO. Return the number of bytes flushed.

This function is only defined for BufWriter, not generically for AbstractBufWriter. Generic code should use flush instead.

julia> io = IOBuffer();

julia> wtr = BufWriter(io);

julia> write(wtr, "hello!");

julia> take!(io)
UInt8[]

julia> shallow_flush(wtr)
6

julia> String(take!(io))
"hello!"
BufferIO.take_inner — Function
take_inner(io)

Close io without closing the inner io wrapped by io, and return the inner io.

If io is a writer, any data buffered in io is first written to the inner io, so no written data is lost. The inner io is not flushed.

As with close, io should not be used after calling take_inner, and isopen(io) returns false, unless io remains usable. In particular, operations on io may mutate the wrapped IO, and so may invalidate it. Calling take_inner on an io which is closed, or whose wrapped IO has already been taken, is not allowed.

Implementations may, but are not required to, detect and prevent using io after take_inner, or calling take_inner on a closed io, by throwing an IOError of kind ClosedIO.

Examples

julia> cursor = CursorReader("abc");

julia> reader = IOReader(cursor);

julia> take_inner(reader) === cursor
true
BufferIO.take_inner — Method
take_inner(io::BufReader{T})::T

Close io and return its wrapped IO, without closing the wrapped IO.

Any data buffered in io and not yet consumed is discarded. To keep it, copy it from get_buffer(io) before calling take_inner.

After this, all operations on io which would access the wrapped IO throw an IOError of kind ClosedIO, and close(io) does nothing.

Throw an IOError of kind ClosedIO if io is already closed.

Examples

julia> reader = BufReader(IOBuffer("abcdef"), 4);

julia> String(read(reader, 2))
"ab"

julia> io = take_inner(reader); # "cd" was buffered in reader, and is discarded

julia> String(read(io))
"ef"

julia> read(reader, UInt8)
ERROR: Unsupported operation on closed IO
[...]
BufferIO.take_inner — Method
take_inner(io::BufWriter{T})::T

Write the buffered data of io to its wrapped IO using shallow_flush, then close io and return the wrapped IO. The wrapped IO is neither closed not flushed.

After this, all operations on io which would access the wrapped IO throw an IOError of kind ClosedIO, and close(io) does nothing.

Throw an IOError of kind ClosedIO if io is already closed.

Examples

julia> buf = IOBuffer(); writer = BufWriter(buf);

julia> write(writer, "abc");

julia> take_inner(writer) === buf
true

julia> String(take!(buf)) # buffered data was written
"abc"

julia> flush(writer)
ERROR: Unsupported operation on closed IO
[...]
BufferIO.take_inner — Method
take_inner(io::ExpandReader{T})::T

Close io and return its inner reader, without closing the inner reader.

Any data buffered in io is consumed, and the returned inner reader is positioned at the position of io after the consumption. To keep the data that would be consumed by take_inner, copy it from get_buffer(io) before calling take_inner.

After this, all operations on io which would access the inner reader throw an IOError of kind ClosedIO, and close(io) does nothing.

Throw an IOError of kind ClosedIO if io is already closed.

Examples

julia> reader = ExpandReader(BufReader(IOBuffer("abcdef"), 4));

julia> String(read(reader, 2))
"ab"

julia> kept = String(get_buffer(reader))
"cd"

julia> inner = take_inner(reader); # "cd" is consumed

julia> String(read(inner)) # inner continues right after `kept`
"ef"

julia> read(reader, UInt8)
ERROR: Unsupported operation on closed IO
[...]
BufferIO.take_inner — Method
take_inner(io::ExpandWriter{T})::T

Write the data buffered in the own buffer of io to its inner writer, then close io and return the inner writer. The inner writer is neither closed nor flushed.

After this, all operations on io which would access the inner writer throw an IOError of kind ClosedIO, and close(io) does nothing.

Throw an IOError of kind ClosedIO if io is already closed, or of kind WriterFull if the inner writer can't hold the data buffered in io. In the latter case, io is not closed.

Examples

julia> using BufferIO.Testing: BoundedWriter

julia> sink = UInt8[]; inner = BoundedWriter(sink, 4);

julia> writer = ExpandWriter(inner);

julia> write_repeated(writer, UInt8('a'), 3); fill!(get_nonempty_buffer(writer, 6), UInt8('b'));

julia> consume(writer, 6);

julia> take_inner(writer) === inner
true

julia> flush(inner); String(sink)
"aaabbbbbb"

julia> write(writer, 0x01)
ERROR: Unsupported operation on closed IO
[...]
BufferIO.unsafe_read_all! — Method
unsafe_read_all!(io::AbstractBufReader, p::Ptr{UInt8}, nbytes::Int)::Int

Copy bytes from io to the memory pointed to by p until either nbytes bytes are copied or io is EOF, returning the number of bytes copied. This is the unsafe counterpart of read_all!.

Throw an ArgumentError if nbytes is negative.

See also: unsafe_read, which throws if io is EOF before nbytes bytes are copied.

Warning

Safety: The caller is responsible for ensuring that p is valid, GC protected, and points to at least nbytes bytes of writeable memory.

Examples

julia> reader = CursorReader("abcdef");

julia> v = zeros(UInt8, 8);

julia> GC.@preserve v unsafe_read_all!(reader, pointer(v), 2)
2

julia> GC.@preserve v unsafe_read_all!(reader, pointer(v, 3), 6) # 4 bytes remaining
4

julia> String(v[1:6])
"abcdef"
BufferIO.write_repeated — Method
write_repeated(io::AbstractBufWriter, byte::UInt8, n::Int)::Int

Write byte to io n times, and return n. This is equivalent to write(io, fill(byte, n)), but more efficient.

Throw an InexactError if n is negative. If io cannot make room for all n bytes, throw an IOError of kind IOErrorKinds.WriterFull. In that case, some bytes may already have been written.

Examples

julia> w = VecWriter(collect(b"abc"));

julia> write_repeated(w, UInt8('x'), 6)
6

julia> takestring!(w)
"abcxxxxxx"
BufferIO.IOErrorKinds.BufferTooShort — Constant
BufferTooShort

Thrown by various functions that require a minimum buffer size, which the io cannot provide. This should only be thrown if the buffer is unable to grow to the required size, and not if e.g. the buffer does not expand because the io is EOF.

BufferIO.IOErrorKinds.WriterFull — Constant
WriterFull

Occurs when a writer cannot make room for more data, i.e. when grow_buffer returns zero, such that a write operation cannot complete. If a writer can make room, but not a contiguous buffer of a required minimum size, BufferTooShort is thrown instead.

Base.readavailable — Method
readavailable(io::AbstractBufReader)::Vector{UInt8}

Read the available bytes of io to a new Vector{UInt8}, except if zero bytes are available. In that case, it will attempt to get more bytes exactly once. If still no bytes are available, io is EOF, and the resulting vector is empty.

Examples

julia> io = BufReader(IOBuffer("hello, world!"), 10);

julia> bytesavailable(io)
0

julia> readavailable(io) == b"hello, wor"
true

julia> readavailable(io) == b"ld!"
true

julia> readavailable(io) == b""
true
Base.peek — Method
peek(io::AbstractBufReader, UInt8)::UInt8

Get the next UInt8 in io, without advancing the IO, or throw an IOError containing IOErrorKinds.EOF if io is EOF.

Examples

julia> io = CursorReader("x");

julia> peek(io, UInt8) === UInt8('x')
true

julia> read(io, UInt8) == UInt8('x') # now EOF
true

julia> peek(io, UInt8)
ERROR: End of file
[...]
Base.read — Method
read(io::AbstractBufReader, UInt8)::UInt8

Get the next UInt8 in io, or throw an IOError containing IOErrorKinds.EOF if io is EOF.

Examples

julia> io = CursorReader("xy");

julia> read(io, UInt8) === UInt8('x')
true

julia> read(io, UInt8) === UInt8('y')
true

julia> read(io, UInt8)
ERROR: End of file
[...]
Base.readbytes! — Function
readbytes!(io::AbstractBufReader, b::AbstractVector{UInt8}, nb::Integer=length(b))::Int

Read at most nb bytes from io into b, returning the number of bytes read. If nb > 0, this function will read zero bytes if and only if io is EOF.

b must use one-based indexing. The size of b will be increased if needed (i.e. if nb is greater than length(b) and enough bytes could be read), but it will never be decreased.

It is generally preferred to use read_into! instead of this method.

Base.unsafe_read — Method
unsafe_read(io::AbstractBufReader, p::Ptr{UInt8}, nbytes::Integer)::Nothing

Copy exactly nbytes bytes from io to the memory pointed to by p. If io reaches EOF before nbytes bytes are copied, throw an IOError of kind IOErrorKinds.EOF. In that case, the bytes before EOF may already have been copied.

Throw an InexactError if nbytes is negative.

See also: unsafe_read_all!, which stops at EOF instead of throwing.

Warning

Safety: The caller is responsible for ensuring that p is valid, GC protected, and points to at least nbytes bytes of writeable memory.

Examples

julia> reader = CursorReader("abcdef");

julia> v = zeros(UInt8, 4);

julia> GC.@preserve v unsafe_read(reader, pointer(v), 4)

julia> String(v)
"abcd"

julia> GC.@preserve v unsafe_read(reader, pointer(v), 4) # 2 bytes remaining
ERROR: End of file
[...]
Base.copyline — Method
copyline(
    out::Union{IO, AbstractBufWriter},
    from::AbstractBufReader;
    keep::Bool = false
) -> out

Copy one line from from to out, returning out. A line is defined as data up to and including \n (byte 0x0a), or all remaining data in from if no such byte is present.

If keep is false, as it is by default, the trailing \r\n (encoded as 0x0d 0x0a) or \n will not be copied to out, but it will be consumed from from.

Base.close — Method
close(io::Union{AbstractBufReader, AbstractBufWriter})

Close io, releasing its underlying resources, such as a wrapped IO or a file handle. If io is a writer, close first flushes io, so no written data is lost.

Closing is idempotent: Closing an io which is already closed does nothing.

After io is closed, the behaviour of operations on io is unspecified, except that close and isopen behave as documented, and that no operation causes undefined behaviour. Depending on the implementation, operations on a closed io may throw an IOError of kind ClosedIO, or continue to work normally.

See also: take_inner

Examples

julia> reader = BufReader(IOBuffer("abc"));

julia> close(reader); close(reader) # closing twice does nothing

julia> isopen(reader)
false

julia> read(reader)
ERROR: Unsupported operation on closed IO
[...]
Base.isopen — Method
isopen(io::Union{AbstractBufReader, AbstractBufWriter})::Bool

Return whether io is still usable, that is, return false if and only if io has been made unusable by close or take_inner, or a similar function.

For types that remain usable after close, such as VecWriter, isopen should return true, even after close has been called on the io. Hence, isopen(io) does not generically tell whether close(io) has been called.

Examples

julia> reader = BufReader(IOBuffer("abc"));

julia> isopen(reader)
true

julia> close(reader); isopen(reader)
false

julia> cursor = CursorReader("abc");

julia> close(cursor); isopen(cursor) # closing a CursorReader does nothing
true
Base.filesize — Method
filesize(io::AbstractBufReader)::Int

Get the total size, in bytes, which can be read by io, and the span in which io can be seeked. Types implementing filesize should also implement seek.

The filesize does not depend on the current reading state of the io, i.e. reading bytes should not change the filesize.

Base.filesize — Method
filesize(io::AbstractBufWriter)::Int

Get the size, in bytes, of the data written to io, including data which has been committed with consume, but not yet flushed. This is the size the underlying file or IO will have once io is flushed, provided nothing else modifies it.

0:filesize(io) is the range of valid offsets for seek, and position(io) is always in this range. Writing beyond the end of the data increases the filesize. Other operations, including seeking and flushing, do not change it.

Types implementing filesize should also implement seek and position.

Examples

julia> io = BufWriter(IOBuffer());

julia> write(io, "abc");

julia> filesize(io) # includes unflushed data
3

julia> seek(io, 1); filesize(io)
3

julia> write(io, "xyz"); filesize(io)
4
Base.seek — Method
seek(io::AbstractBufReader, offset::Integer) -> io

Seek io to the zero-based position offset, if io supports seeking, and return io. When offset === 0, this is equivalent to seekstart. If filesize(io) is implemented, seek(io, filesize(io)) is equivalent to seekend(io).

Valid offsets are 0:filesize(io), if io implements filesize. Seeking outside these bounds throws an IOError of kind BadSeek.

This method is not generically defined for AbstractBufReader.

julia> rdr = BufReader(IOBuffer("Hello, world!"));

julia> String(read(rdr, 5))
"Hello"

julia> seek(rdr, 3);

julia> String(read(rdr, 5))
"lo, w"

julia> seek(rdr, 13);

julia> read(rdr)
UInt8[]
Base.seek — Method
seek(io::AbstractBufWriter, offset::Integer) -> io

Seek io to the zero-based position offset, and return io. Subsequent writes write (or overwrite) data starting at offset.

Valid values for offset are in 0:filesize(io), if filesize(io) is defined. Seeking outside these bounds throws an IOError of kind BadSeek, and leaves io unchanged.

Seeking never changes the filesize, nor any data already written to io, including data not yet flushed. In particular, seeking never truncates the underlying file or IO.

Seeking may write buffered data to the underlying IO, but need not flush it. Use flush for that.

This method is not generically defined for AbstractBufWriter. Implementors of seek should also define filesize(io) and position(io)

Examples

julia> buf = IOBuffer();

julia> io = BufWriter(buf);

julia> write(io, "abcdef");

julia> seek(io, 3);

julia> write(io, "xyzghi");

julia> seek(io, 10)
ERROR: Invalid seek, possibly seek out of bounds
[...]

julia> flush(io);

julia> String(take!(buf))
"abcxyzghi"
Base.skip — Method
skip(io::AbstractBufReader, offset::Integer) -> io

Seek io by offset relative to its current position, and return io.

Like relative_seek, this requires io to support seeking, and seeking beyond either end of io throws an IOError of kind BadSeek. To advance a reader that may not support seeking, by reading and discarding bytes, use discard_all or discard_exact.

Examples

julia> reader = CursorReader("abcdef");

julia> skip(reader, 4); String(read(reader, 1))
"e"

julia> skip(reader, -3); String(read(reader))
"cdef"

julia> skip(reader, 1)
ERROR: Invalid seek, possibly seek out of bounds
[...]
Base.position — Method
Base.position(io::AbstractBufWriter)::Int

Get the zero-based stream position.

If the stream position is p (zero-based), then the next byte written will be byte number p + 1 (one-based) in the file. The stream position accounts for buffered (consumed, but unflushed) bytes. The value of position is always in 0:filesize(io), if filesize is defined.

Examples

julia> io = BufWriter(IOBuffer());

julia> write(io, "hello");

julia> (filesize(io), position(io))
(5, 5)

julia> seek(io, 2);

julia> (filesize(io), position(io))
(5, 2)
Base.position — Method
Base.position(io::AbstractBufReader)::Int

Get the zero-based stream position.

If the stream position is p (zero-based), then the next byte read will be byte number p + 1 (one-based). The value of position must be in 0:filesize(io), if filesize is defined.

Examples

julia> reader = CursorReader("abcdefghij");

julia> position(reader) # zero-indexed
0

julia> read(reader, 3) |> String
"abc"

julia> position(reader)
3

julia> read(reader, 2) |> String
"de"

julia> position(reader)
5

julia> seekstart(reader); position(reader)
0

Testing utilities

BufferIO.Testing — Module
module BufferIO.Testing

Minimal implementations of AbstractBufReader and AbstractBufWriter intended for testing code that is generic over buffered readers and writers, similar in spirit to Test.GenericString.

The types in this module implement only the required interface methods, and have deliberately restricted buffers, or buffers which are filled in controlled increments. This exercises the code paths that BufReader or VecWriter never hit, such as fill_buffer returning nothing, and fallback definitions of optional methods. Furthermore, the readers overwrite their old buffer whenever views of their buffer may be invalidated, which helps to catch code that uses such views after they have been invalidated.

These types are not optimized for performance, and should only be used in tests.

See also: BoundedReader, ChunkedReader, BoundedWriter

BufferIO.Testing.BoundedReader — Type
BoundedReader(data, max_buffer_size::Int; max_fill::Int = 1) <: AbstractBufReader

Reader of the bytes of data, whose buffer can hold at most max_buffer_size bytes, and where each call to fill_buffer adds at most max_fill bytes. data can be any object x which implements ImmutableMemoryView{UInt8}(x).

When the buffer holds max_buffer_size bytes, fill_buffer returns nothing, and so BufferIO.is_expandable(BoundedReader) is false. This makes BoundedReader useful to test that generic code correctly handles readers with a limited buffer size, and, with a small max_fill, readers which fill their buffer in small increments.

Views returned by get_buffer are invalidated by fill_buffer, consume and close. To catch code which nonetheless uses such views, every call to these functions moves the buffer to new memory, and overwrites the old memory by flipping all bits of every byte. Hence, invalidated views will appear to contain different data. The bytes of data itself are never modified.

BoundedReader only implements the methods required of an AbstractBufReader, so optional methods such as seek and position are unsupported or use the generic fallback definitions.

Examples

julia> using BufferIO.Testing: BoundedReader

julia> reader = BoundedReader("abcdefg", 4; max_fill = 3);

julia> fill_buffer(reader), fill_buffer(reader), fill_buffer(reader)
(3, 1, nothing)

julia> buffer = get_buffer(reader); String(buffer)
"abcd"

julia> consume(reader, 3); buffer == b"abcd" # buffer invalidated by consume
false

julia> String(read(reader))
"defg"

julia> BufferIO.is_expandable(BoundedReader)
false
BufferIO.Testing.ChunkedReader — Type
ChunkedReader(chunks) <: AbstractBufReader

Reader of the bytes of chunks, where the nth call to fill_buffer adds exactly the bytes of the nth chunk to the buffer, and returns 0 once all chunks have been added. Each chunk can be any object x which implements ImmutableMemoryView{UInt8}(x), and must not be empty, since fill_buffer returning 0 signals EOF.

The buffer can grow without limit, so fill_buffer never returns nothing, and BufferIO.is_expandable(ChunkedReader) is true. This makes ChunkedReader useful to test that generic code correctly handles data which arrives in pieces, by controlling exactly where the buffer ends after each call to fill_buffer, e.g. in the middle of a line, or between a \r and a \n.

Like BoundedReader, every call to fill_buffer, consume and close moves the buffer to new memory, and overwrites the old memory by flipping all bits of every byte, to catch code which uses views of the buffer after they have been invalidated. The bytes of the chunks themselves are never modified.

ChunkedReader only implements the methods required of an AbstractBufReader, so optional methods such as seek and position are unsupported or use the generic fallback definitions.

Examples

julia> using BufferIO.Testing: ChunkedReader

julia> reader = ChunkedReader(["abc\r", "\nde", "f"]);

julia> fill_buffer(reader), fill_buffer(reader)
(4, 3)

julia> String(get_buffer(reader))
"abc\r\nde"

julia> consume(reader, 5); fill_buffer(reader), fill_buffer(reader)
(1, 0)

julia> String(read(reader))
"def"

julia> BufferIO.is_expandable(ChunkedReader)
true
BufferIO.Testing.BoundedWriter — Type
BoundedWriter(sink::Vector{UInt8}, max_buffer_size::Int) <: AbstractBufWriter

Writer with a fixed-size buffer of max_buffer_size bytes, which is never expanded. When flushed, or when grow_buffer is called, all buffered bytes are appended to sink.

This makes BoundedWriter useful to test that generic code correctly handles writers with a limited buffer size.

BoundedWriter only implements the methods required of an AbstractBufWriter, so optional methods such as seek and position are unsupported or use the generic fallback definitions. BufferIO.is_expandable(BoundedWriter) is false.

Examples

julia> using BufferIO.Testing: BoundedWriter

julia> sink = UInt8[]; writer = BoundedWriter(sink, 4);

julia> length(get_buffer(writer))
4

julia> write(writer, "Hello, world!")
13

julia> String(copy(sink)) # 12 bytes flushed, 1 still buffered
"Hello, world"

julia> flush(writer); String(sink)
"Hello, world!"