Reference

abstract type AbstractBufReader end

An AbstractBufReader is an IO type that exposes a buffer to the user, thereby allowing efficient IO.

Subtypes T of this type should implement at least:

• get_buffer(io::T)
• fill_buffer(io::T)
• consume(io::T, n::Int)
• Base.close(io::T)

Extended help

Subtypes may optionally define the following methods. See their docstring for BufReader / BufWriter for details of the implementation:

• Base.seek
• Base.filesize

Subtypes T of this type have implementations for many Base IO methods, but with more precisely specified semantics. See docstrings of the specific functions of interest.

abstract type AbstractBufWriter end

An AbstractBufWriter is an IO-like type which exposes mutable memory to the user, which can be written to directly. This can help avoiding intermediate allocations when writing. For example, integers can usually be written to buffered writers without allocating.

Subtypes of this type should not have a zero-sized buffer which cannot expand when calling grow_buffer.

Subtypes T of this type should implement at least:

• get_buffer(io::T)
• grow_buffer(io::T)
• Base.close(io::T)
• Base.flush(io::T)
• consume(io::T, n::Int)

They may optionally implement

• get_unflushed(io::T)
• get_nonempty_buffer(io::T, ::Int)

Extended help

• Methods of Base.close should make sure that calling close on an already-closed object has no visible effects.
• flush(x::T) should be implemented, but may simply return nothing if there is no underlying stream to flush to.

BufReader(f, io::IO, [buffer_size::Int])

Create a BufReader wrapping io, then call f on the BufReader, and close the reader once f is finished or if it errors.

This pattern is useful for automatically cleaning up the resource of io.

julia> io = IOBuffer("hello world");

julia> BufReader(io) do reader
           println(String(read(io, 5)))
       end
hello

julia> read(io) # closing reader closes io also
ERROR: ArgumentError: read failed, IOBuffer is not readable
[...]

BufReader{T <: IO} <: AbstractBufReader
BufReader(io::IO, [buffer_size::Int])::BufReader

Wrap an IO in a struct with a new buffer, giving it the AbstractBufReader interface.

The BufReader has an infinitely growable buffer, and will only grow the buffer if fill_buffer is called while its internal buffer is full.

Throw an ArgumentError if buffer_size is less than 1.

julia> rdr = BufReader(IOBuffer("Hello, world!\nabc\r\ndef"));

julia> get_buffer(rdr)
0-element MemoryViews.ImmutableMemoryView{UInt8}

julia> peek(rdr)
0x48

julia> readline(rdr)
"Hello, world!"

julia> String(readavailable(rdr))
"abc\r\ndef"

BufWriter(f, io::IO, [buffer_size::Int])

Create a BufWriter wrapping io, then call f on the BufWriter, and close the writer once f is finished or if it errors.

This pattern is useful for automatically cleaning up the resource of io.

julia> io = IOBuffer();

julia> BufWriter(io) do writer
           write(writer, "hello, world!")
           shallow_flush(writer)
           seekstart(io)
           println(String(read(io)))
       end
hello, world!

julia> iswritable(io) # closing reader closes io also
false

BufWriter{T <: IO} <: AbstractBufWriter
BufWriter(io::IO, [buffer_size::Int])::BufWriter

Wrap an IO in a struct with a new buffer, giving it the AbstractBufWriter interface.

The BufWriter has an infinitely growable buffer, and will only expand the buffer if grow_buffer is called on it while it does not contain any data (as shown by get_unflushed).

Throw an ArgumentError if buffer_size is < 1.

julia> io = IOBuffer(); wtr = BufWriter(io);

julia> print(wtr, "Hello!")

julia> write(wtr, [0x1234, 0x5678])
4

julia> read(io) # wtr not flushed
UInt8[]

julia> flush(wtr); seekstart(io); String(read(io))
"Hello!4\x12xV"

julia> get_unflushed(wtr)
0-element MemoryViews.MutableMemoryView{UInt8}

CursorReader(x) <: AbstractBufReader

A seekable, stateful reader of the content of any object x which implements MemoryView(x)::MemoryView{UInt8}.

Closing it does nothing.

julia> rdr = CursorReader("some\ncontent\nhere");

julia> readline(rdr)
"some"

julia> read(rdr, String)
"content\nhere"

julia> seek(rdr, 8);

julia> read(rdr, String)
"tent\nhere"

IOError

This type is thrown by errors of AbstractBufReader. They contain the .kind::IOErrorKind public property.

See also: IOErrorKinds.IOErrorKind

Examples

julia> rdr = CursorReader("some content");

julia> try
           seek(rdr, 500)
       catch error
           if error.kind == IOErrorKinds.BadSeek
               println(stderr, "Seeking operation out of bounds")
           else
               rethrow()
           end
        end
Seeking operation out of bounds

IOReader{T <: AbstractBufReader} <: IO

Wrapper type to convert an AbstractBufReader to an IO.

IOReaders implement the same part of the IO interface as AbstractBufReader, so this type is only used to satisfy type constraints.

Examples

julia> io = CursorReader("hello");

julia> f(x::IO) = String(read(x));

julia> f(io)
ERROR: MethodError: no method matching f(::CursorReader)
[...]

julia> f(IOReader(io))
"hello"

VecWriter([vec::Vector{UInt8}]) <: AbstractBufWriter

Create an AbstractBufWriter backed by a Vector{UInt8}. Read the (public) property .vec to get the vector back.

This type is useful as an efficient string builder through String(io.vec) or takestring!(io) (the latter in Julia 1.13+).

Functions flush and close do not affect the writer.

Mutating io will mutate vec and vice versa. Neither vec nor io will be invalidated by mutating the other, but doing so may affect the implicit (non-semantic) behaviour (e.g. memory reallocations or efficiency) of the other. For example, repeated and interleaved push!(vec) and write(io, x) may be less efficient, if one operation has memory allocation patterns that is suboptimal for the other operation.

julia> vw = VecWriter();

julia> write(vw, "Hello, world!", 0xe1fa)
15

julia> append!(vw.vec, b"More data");

julia> String(vw.vec)
"Hello, world!\xfa\xe1More data"

consume(io::Union{AbstractBufReader, AbstractBufWriter}, n::Int)::Nothing

Remove the first n bytes of the buffer of io. Consumed bytes will not be returned by future calls to get_buffer.

If n is negative, or larger than the current buffer size, throw an IOError with ConsumeBufferError kind. This check is a boundscheck and may be elided with @inbounds.

fill_buffer(io::AbstractBufReader)::Union{Int, Nothing}

Fill more bytes into the buffer from io's underlying buffer, returning the number of bytes added. After calling fill_buffer and getting n, the buffer obtained by get_buffer should have n new bytes appended.

This function must fill at least one byte, except

• If the underlying io is EOF, or there is no underlying io to fill bytes from, return 0
• If the buffer is not empty, and cannot be expanded, return nothing.

Buffered readers which do not wrap another underlying IO, and therefore can't fill its buffer should return 0 unconditionally. This function should never return nothing if the buffer is empty.

get_buffer(io::AbstractBufReader)::ImmutableMemoryView{UInt8}

Get the available bytes of io.

Calling this function when the buffer is empty should do actual system I/O, and in particular should not attempt to fill the buffer. To fill the buffer, call fill_buffer.

get_buffer(io::AbstractBufWriter)::MutableMemoryView{UInt8}

Get the available mutable buffer of io that can be written to.

Calling this function should do actual system I/O, and in particular should not attempt to flush data from the buffer. To increase the size of the buffer, call grow_buffer.

get_nonempty_buffer(x::AbstractBufReader)::Union{Nothing, ImmutableMemoryView{UInt8}}

Get a buffer with at least one byte, if bytes are available. Otherwise, fill the buffer, and return the newly filled buffer. Returns nothing only if x is EOF.

get_nonempty_buffer(x::AbstractBufWriter)::Union{Nothing, MutableMemoryView{UInt8}}

Get a buffer with at least one byte, if bytes are available. Otherwise, call grow_buffer, then get the buffer again. Returns nothing if the buffer is still empty.

get_nonempty_buffer(
    io::AbstractBufWriter, min_size::Int
)::Union{Nothing, MutableMemoryView{UInt8}}

Get a nonempty buffer of at least size min_size.

This function need not be implemented for subtypes of AbstractBufWriter that do not flush their writes to an underlying IO.

get_unflushed(io::AbstractBufWriter)::MutableMemoryView{UInt8}

Return a view into the buffered data already written to io and consumed, but not yet flushed to its underlying IO.

Bytes not appearing in the buffer may not be completely flushed if there are more layers of buffering in the IO wrapped by io. However, any bytes already consumed and not returned in get_unflushed should not be buffered in io.

Mutating the returned buffer is allowed, and should not cause io to malfunction. When io flushes, the updated values in the buffer will be flushed.

This function has no default implementation and methods are optionally added to subtypes of AbstractBufWriter that can fullfil the above restrictions.

grow_buffer(io::AbstractBufWriter)::Int

Increase the amount of bytes in the writeable buffer of io if possible, returning the number of bytes added. After calling grow_buffer and getting n, the buffer obtained by get_buffer should have n more bytes.

• If there is data in the buffer of io, flush it to the underlying io if possible.
• Else, if io's buffer can be expanded, do so.
• Else, return zero

line_views(x::AbstractBufReader; chomp::Bool=true)

Create an efficient iterator of lines of x. The returned views are ImmutableMemoryView{UInt8} views into x's buffer, and are invalidated when x is mutated or the line iterator is advanced.

A line is defined as all data up to and including \n (0x0a) or \r\n (0x0d 0x0a), or the remainder of the data in io if no \n byte was found. If the input is empty, this iterator is also empty.

The lines are iterated as ImmutableMemoryView{UInt8}. Use the package StringViews.jl to turn them into AbstractStrings. The chomp keyword (default: true), controls whether any trailing \r\n or \n should be removed from the output.

If x had a limited buffer size, and an entire line cannot be kept in the buffer, an ArgumentError is thrown.

The resulting iterator will NOT close x when done, this must be handled by the caller.

read_all!(io::AbstractBufReader, dst::MutableMemoryView{UInt8})::Int

Read bytes into dst until either dst is filled or io is EOF, returning the number of bytes read.

read_into!(x::AbstractBufReader, dst::MutableMemoryView{UInt8})::Int

Read bytes into the beginning of dst, returning the number of bytes read. This function will always read at least 1 byte, except when dst is empty, or x is EOF.

This function is defined generically for AbstractBufReader. New methods should strive to do at most one read call to the underlying IO, if x wraps such an IO.

resize_buffer(io::Union{BufWriter, BufReader}, n::Int) -> io

Resize the internal buffer of io to exactly n bytes.

Throw an ArgumentError if n is less than 1, or lower than the currently number of buffered bytes (length of get_unflushed for BufWriter, length of get_buffer for BufReader).

julia> w = BufWriter(IOBuffer());

julia> write(w, "abc")
3

julia> length(get_buffer(resize_buffer(w, 5)))
2

julia> resize_buffer(w, 2)
ERROR: ArgumentError: Buffer size smaller than current number of buffered bytes
[...]

julia> shallow_flush(w)
3

julia> resize_buffer(w, 2) === w
true

shallow_flush(io::AbstractBufWriter)::Int

Clear the buffer(s) of io by writing to the underlying I/O, but do not flush the underlying I/O. Return the number of bytes flushed.

This function is not generically defined for AbstractBufReader.

julia> io = IOBuffer();

julia> wtr = BufWriter(io);

julia> write(wtr, "hello!");

julia> take!(io)
UInt8[]

julia> shallow_flush(wtr)
6

julia> String(take!(io))
"hello!"

skip_exact(io::AbstractBufReader, n::Integer)::Nothing

Like skip, but throw an IOError of kind IOErrorKinds.EOF if n bytes could not be skipped.

See also: Base.skip

IOErrorKind

Enum indicating what error was thrown. The current list is non-exhaustive, and more may be added in future releases. The integral value of these enums are subject to change in minor versions.

Current errors:

• ConsumeBufferError: Occurs when calling consume with a negative amount of bytes, or with more bytes than length(get_buffer(io))
• EOF: Occurs when trying a reading operation on a file that has reached end-of-file
• BufferTooShort: Thrown by various functions that require a minimum buffer size, which the io cannot provide. This should only be thrown if the buffer is unable to grow to the required size, and not if e.g. the buffer does not expand because the io is EOF.
• BadSeek: An out-of-bounds seek operation was attempted
• PermissionDenied: Acces was denied to a system (filesystem, network, OS, etc.) resource
• NotFound: Resource was not found, e.g. no such file or directory
• BrokenPipe: The operation failed because a pipe was broken. This typically happens when writing to stdout or stderr, which then gets closed.
• AlreadyExists: Resource (e.g. file) could not be created because it already exists
• NotADirectory: Resource is unexpectedly not a directory. E.g. a path contained a non-directory file as an intermediate component.
• IsADirectory: Resource is a directory when a non-directory was expected
• DirectoryNotEmpty: Operation cannot succeed because it requires an empty directory
• InvalidFileName: File name was invalid for platform, e.g. too long name, or invalid characters.

module IOErrorKinds

Used as a namespace for IOErrorKind.

unsafe_read(io::AbstractBufReader, ref, nbytes::UInt)::Int

Copy nbytes from io into ref, returning the number of bytes copied. If io reached end of file, stop at EOF. ref is converted to a pointer using Base.unsafe_convert(Ptr{UInt8}, Base.cconvert(Ptr, ref)).

Safety: The user must ensure that the resulting pointer is valid, and points to at least nbytes of writeable memory.

readavailable(io::AbstractBufReader)::Vector{UInt8}

Read the available bytes of io to a new Vector{UInt8}, except if zero bytes are available. In that case, it will attempt to get more bytes exactly once. If still no bytes are available, io is EOF, and the resulting vector is empty.

peek(io::AbstractBufReader)::UInt8

Get the next UInt8 in io, without advancing io, or throw an IOError containing IOErrorKinds.EOF if io is EOF.

read(io::AbstractBufReader, UInt8)::UInt8

Get the next UInt8 in io, or throw an IOError containing IOErrorKinds.EOF if io is EOF.

readbytes!(io::AbstractBufReader, b::AbstractVector{UInt8}, nb::Integer=length(b))::Int

Read at most nb bytes from io into b, returning the number of bytes read. This function will read zero bytes if and only if io is EOF.

b must use one-based indexing. The size of b will be increased if needed (i.e. if nb is greater than length(b) and enough bytes could be read), but it will never be decreased.

It is generally preferred to use read_into! instead of this method.

copyline(
    out::Union{IO, AbstractBufWriter},
    from::AbstractBufReader;
    keep::Bool = false
) -> out

Copy one line from from to out, returning out. A line is defined as data up to and including \n (byte 0x0a), or all remaining data in from if no such byte is present.

If keep is false, as it is by default, the trailing \r\n (encoded as 0x0d 0x0a) or \n will not be copied to out, but it will be consumed from from.

This function may throw an IOerror with IOErrorKinds.BufferTooShort, if all the following occurs:

• keep is false
• The reader has a buffer size of 1
• The reader cannot expand its buffer
• The only byte in the buffer is \r (0x0d).

filesize(io::AbstractBufReader)::Int

Get the total size, in bytes, which can be read by io, and the span in which io can be seeked. Types implementing filesize should also implement seek.

The filesize does not depend on the current reading state of the io, i.e. reading bytes should not change the filesize.

filesize(x::AbstractBufWriter)::Int

Get the total size, in bytes, of data written to io. This includes previously flushed data, and data comitted by consume but not flushed. Types implementing filesize should also implement seek.

seek(io::AbstractBufReader, offset::Int) -> io

Seek io to the zero-based position offset, if io supports seeking, and return io. When offset === 0, this is equivalent to seekstart. If filesize(io) is implemented, this is equivalent to seekend(io).

Valid offsets are 0:filesize(io), if io implements filesize. Seeking outside these bounds throws an IOError of kind BadSeek.

This method is not generically defined for AbstractBufReader.

julia> rdr = BufReader(IOBuffer("Hello, world!"));

julia> String(read(rdr, 5))
"Hello"

julia> seek(rdr, 3);

julia> String(read(rdr, 5))
"lo, w"

julia> seek(rdr, 13);

julia> read(rdr)
UInt8[]

seek(io::AbstractBufWriter, offset::Int) -> io

Seek io to the zero-based position offset.

Valid values for offset are in 0:filesize(io), if filesize is defined. Seeking outside these bounds throws an IOError of kind BadSeek.

If seeking to before the current position (as defined by position), data between the new and the previous position need not be changed, and the underlying file or IO need not immediately be truncated. However, new write operations should write (or overwrite) data at the new position.

This method is not generically defined for AbstractBufReader. Implementors of seek should also define filesize(io) and position(io)

skip(io::AbstractBufReader, n::Integer)::Int

Read n bytes from io, or until EOF, whichever comes first, and discard the read bytes. Return the number of bytes read.

This function is defined generically for AbstractBufReader by reading bytes, not by seeking. Subtypes of AbstractBufReader may implement this using seeking. In order to skip a generic AbstractBufReader and guarantee seeking is used, use seek(io, position(io) + n).

Throws an ArgumentError if n < 0.

See also: skip_exact