ValueOnly Binary Format

A technical overview of the ValueOnly binary format used by Wiener.

What Is ValueOnly?

Conventional binary formats and JSON store key names alongside data.

{ "item_id": 1001, "name": "Sword", "hp": 0, "attack": 45 }

Wiener's ValueOnly format stores no key names. Instead, the C#/C++ code generated at convert time records each field's byte offset, and property accesses go directly to that offset.

How the Speed Works

JSON / MessagePack:
  Receive data → parse → match key names → get value

ValueOnly:
  Receive data → calculate offset → get value  (zero key matching)

Because key name matching never occurs, access time is constant regardless of data size. This is the core design that enables 0.1-second loads with 2 million cells.

Contribution to Security

Since key names do not exist in the binary, opening the file in a binary editor reveals no field names. Combined with the obfuscation feature, this provides an additional layer of protection.

Relationship to Generated Code

ValueOnly works as a pair with the code Wiener generates.

// Example of Wiener-generated code
public class ItemData : IWienerDictionaryData<int>
{
    public int ItemId { get; private set; }
    public string Name { get; private set; }
    public int Attack { get; private set; }

    public void Read(WienerDataReader reader)
    {
        ItemId = reader.ReadInt32();
        Name   = reader.ReadString();
        Attack = reader.ReadInt32();
    }
}

The reader streams through the binary from the start, advancing its internal offset by the size of each field read. Each ReadXXX call simply reads the next field and moves forward — no key name matching, no seeking.

The normal LoadBasicPack path assumes that the generated source code and basic.bytes have exactly the same structure. This is the release-oriented path: it avoids extra checks and loads as fast as possible.

During development, however, the generated code and binary can temporarily drift apart, for example when YAML or generated code has been updated but an older basic.bytes is still being loaded. In that case, the debug-only LoadBasicPackFromHash method can load data into variables whose binary hashes match, even when the overall structure is not a complete match. Variables missing from the binary use default values, and data that exists in the binary but not in the source is ignored.

LoadBasicPackFromHash is a safety net for development and verification, and it is slower than the normal load path. For release builds, generate source code and binary data from the same conversion result and use LoadBasicPack for exact-match loading.

Limitations

ValueOnly cannot be read by tools other than Wiener. This is a deliberate design trade-off: speed and security over portability.

For debugging or sharing data with other systems, use text-based output formats depending on the purpose. Use CSV when you want output close to the raw Excel or Google Sheets data, SQL (CSV) when you want only the fields selected by YAML, and JSON for general-purpose data integration.

Last Updated::
Contributors: artisan-sawasaka