Module encoding/utf8
UTF-8 primitives — the single decoder/encoder the rest of std is built on.
Everything here is byte-level and allocation-free unless it is handed a
buffer to append to. decode/decode_parts validate to RFC 3629: overlong
forms, UTF-16 surrogates (U+D800..U+DFFF) and scalars above U+10FFFF are
rejected, not silently accepted.
Layering
This module deliberately imports only std/string/rune and
std/collections/array_list. It must stay below std/error and std/fmt,
because std/string/string.yo — which both of those import — is itself a
consumer. That is also why Utf8Error carries an inherent message()
returning str instead of a ToString/Error() impl: those traits live
above this layer. Callers that need a throwable error wrap it in their own
module's error enum (StringError.InvalidUtf8 does exactly that).
Example
{ decode, encode_into, validate } :: import "std/encoding/utf8";
{ ArrayList } :: import "std/collections/array_list";
bytes := `héllo`.as_bytes();
match(validate(bytes), .Ok(_) => (), .Err(e) => println(e.message()));
// Walk the string one rune at a time.
i := usize(0);
while(i < bytes.len(), {
match(decode(bytes, i), .Ok(d) => { i = (i + d.width); }, .Err(_) => { i = (i + usize(1)); });
});
Stability
stable. Three things make that a real commitment rather than a default.
It has not moved since the D8 consolidation (#286) folded four scattered
decoders into this one; every change since has been documentation. It has
the tree's widest internal consumer set — std/string/string.yo,
rune.yo, string_builder.yo, unicode.yo, std/imm/string.yo,
std/fmt/to_string.yo, std/regex/{index,parser,vm}.yo and
std/encoding/{json,utf16}.yo all sit on it — so a signature change here
is a signature change to String itself, and the 35 tests in
tests/encoding/utf8.test.yo pin the RFC 3629 rejections (overlong
forms, surrogates, above U+10FFFF) that everything above depends on.
And the names already have the shape D2 wants for the rest of this
directory: utf8.decode / utf8.encode / utf8.validate on the imported
module value, with no utf8_ prefix repeating the module. The
module-prefix rename that keeps json, base64, hex, percent and
toml unstable does not apply here, because this module is already the
target shape.
Growth is additive by construction: a new query over bytes is a new
function, and the Layering note above is what keeps it that way — this
module cannot grow a dependency on std/error or std/fmt without
breaking the cycle that std/string sits in. The one thing a freeze does
NOT promise is Utf8Error's inherent message() -> str staying the only
rendering channel; adding ToString / Error() impls if the layering
ever permits it is additive, and a caller matching the variants is
unaffected either way.
Types
Why a byte sequence is not valid UTF-8.
Every variant carries index, the byte offset the offending sequence
STARTS at, so a caller can report the position without re-scanning.
Variants
| Variant | Fields | Description |
|---|---|---|
InvalidStart | index: usize, byte: u8 | The byte at |
UnexpectedEnd | index: usize, expected: usize, available: usize | The sequence starting at |
InvalidContinuation | index: usize, offset: usize, byte: u8 | Byte number |
Overlong | index: usize, code: u32 | The sequence at |
Surrogate | index: usize, code: u32 | The sequence at |
OutOfRange | index: usize, code: u32 | The sequence at |
impl(Utf8Error, ...)
message : (Utf8Error) fn(self : Utf8Error) -> strA short, allocation-free description of the failure.
Returns str rather than String on purpose — see the module header on
layering. Callers that want a formatted message with the offending bytes
build it themselves from the variant's fields.
Parameters
| Name | Type | Notes |
|---|---|---|
self | Utf8Error |
Returns: str
index : (Utf8Error) fn(self : Utf8Error) -> usizeThe byte offset the offending sequence starts at.
Parameters
| Name | Type | Notes |
|---|---|---|
self | Utf8Error |
Returns: usize
One decoded scalar value plus the width of the sequence that carried it.
width is what a chars() / char_indices() iterator advances by, so a
decode step is decode(bytes, i) then i = i + width.
Fields
| Name | Type | Description |
|---|---|---|
code | rune | The decoded scalar value. |
width | usize | Bytes the sequence occupied — 1, 2, 3 or 4. |
Functions
True when b is a UTF-8 continuation byte (0b10xxxxxx).
Parameters
| Name | Type | Notes |
|---|---|---|
b | u8 |
Returns: bool
True when b sits on a rune boundary — i.e. it is not a continuation byte.
This is the predicate for scanning backwards to the start of the rune containing a byte index, and for counting runes by counting boundaries.
Parameters
| Name | Type | Notes |
|---|---|---|
b | u8 |
Returns: bool
Byte width of the UTF-8 sequence that first begins, or 0 when first
cannot begin one.
0 is returned for continuation bytes (0x80..0xBF), for the two
always-overlong leads 0xC0/0xC1, and for 0xF5..0xFF (which could
only spell scalars above U+10FFFF). Callers that must make forward progress
over invalid input treat 0 as "skip one byte".
Parameters
| Name | Type | Notes |
|---|---|---|
first | u8 |
Returns: usize
Bytes to advance by when walking runes FORWARD from b: sequence_len
clamped to 1, so a byte that cannot start a sequence still makes progress.
This is what a tolerant scanner steps by; sequence_len is the strict
answer and returns 0 for "not a lead byte". Every rune-walking loop in
std uses this, which is why it is a function and not five copies of a
cond table.
Parameters
| Name | Type | Notes |
|---|---|---|
b | u8 |
Returns: usize
Bytes r occupies once UTF-8 encoded — 1, 2, 3 or 4.
Parameters
| Name | Type | Notes |
|---|---|---|
r | rune |
Returns: usize
Assemble and validate one UTF-8 sequence from up to four ALREADY-FETCHED bytes.
This is the single implementation decode runs on. It exists as its own
entry point for buffers that are not an ArrayList(u8) — a raw pointer
region, a memory-mapped file — where copying into a list to decode one rune
would be absurd. std/imm/string is exactly that caller.
available is how many of b0..b3 were really readable (1..4); the rest
are ignored. index is only used to build the error, so pass the byte
offset b0 came from.
Parameters
| Name | Type | Notes | Description |
|---|---|---|---|
b0 | u8 | ||
b1 | u8 | ||
b2 | u8 | ||
b3 | u8 | ||
available | usize | ||
index | usize | The byte offset the offending sequence starts at. |
Decode the rune starting at byte index of bytes.
The Err case names exactly what is wrong and where; decode_lossy is the
variant for scanners that must keep going regardless.
Parameters
| Name | Type | Notes | Description |
|---|---|---|---|
bytes | ArrayList(u8) | ||
index | usize | The byte offset the offending sequence starts at. |
Decode the rune starting at byte index, substituting U+FFFD REPLACEMENT
CHARACTER with a width of 1 for anything invalid.
Same contract as Go's utf8.DecodeRune: never fails, and always makes
forward progress, so a while loop over a corrupt buffer still terminates.
Parameters
| Name | Type | Notes | Description |
|---|---|---|---|
bytes | ArrayList(u8) | ||
index | usize | The byte offset the offending sequence starts at. |
Returns: Decoded
Append the UTF-8 encoding of r to out, returning how many bytes were
pushed (1..4).
Infallible for a rune that came from rune.from_u32 or from a decoder in
this module — those are scalar values by construction, so there is nothing
to reject. rune is a plain struct with a public char, so a hand-built
non-scalar (a surrogate half, or a value above U+10FFFF) still encodes to
the bit pattern its bits ask for; use encode_lossy_into when the code
point came from somewhere that can produce one.
Parameters
| Name | Type | Notes |
|---|---|---|
r | rune | |
out | ArrayList(u8) |
Returns: usize
Append the UTF-8 encoding of the raw code point code to out,
substituting U+FFFD when code is not a scalar value (a surrogate half, or
above U+10FFFF). Returns how many bytes were pushed.
This is the entry point for decoders whose intermediate value is a u32
that has not been through rune.from_u32 — JSON \uXXXX escapes, UTF-16
code units, C towlower/towupper results. Substituting keeps the output
buffer valid UTF-8 instead of quietly emitting CESU-8.
Parameters
| Name | Type | Notes | Description |
|---|---|---|---|
code | u32 | The decoded scalar value. | |
out | ArrayList(u8) |
Returns: usize
The UTF-8 encoding of r as a fresh 1..4 byte list.
Parameters
| Name | Type | Notes |
|---|---|---|
r | rune |
Returns: ArrayList(u8)
Check that bytes[from..to) is well-formed UTF-8, on its own — a sequence
that would spill past to is reported as UnexpectedEnd, not accepted.
to is clamped to the buffer length. An empty range is Ok.
Parameters
| Name | Type | Notes |
|---|---|---|
bytes | ArrayList(u8) | |
from | usize | |
to | usize |