Schema registry

A schema registry stores the declared shape of the records flowing through a streaming platform and enforces how that shape may change. The problem it solves is invisible until it hurts: Kafka moves bytes, and nothing in the wire format forces producer and consumer to agree on what those bytes mean. The registry makes the agreement explicit, versioned, and checked at registration time rather than discovered at deserialisation time.

The formats it manages

• Avro - the original registry format; compact binary, schema-driven, with a defined resolution model for reading data written with a different (compatible) schema.
• Protocol Buffers (Protobuf) - schema in .proto files, field numbers as the identity; evolution rules come from the field-number discipline rather than from field names.
• JSON Schema - human-readable and the least compact; useful where the payload must stay inspectable or where the consumers are not JVM-native.

The serialiser on the producer registers the schema, gets back a schema id, and prefixes each record with it - the wire format is a magic byte plus the schema id plus the payload. The consumer reads the id, fetches the schema (caching it), and deserialises against it. The id is how a record carries its own contract.

Compatibility modes: the actual contract

The registry’s rules decide which schema changes are allowed:

• Backward (default): a new schema can read data written by the previous one - consumers upgrade first, producers follow. Adding a field with a default is allowed; removing a required field is not.
• Forward: old schemas can read new data - producers upgrade first.
• Full: both directions - the conservative choice when you cannot control upgrade order.
• None: anything registers; the registry becomes a catalogue, not a gate.

The mode is set per subject (typically one per topic, named <topic>-value or <topic>-key under the default TopicNameStrategy). Choosing it is choosing which side of the producer/consumer pair may change without breaking the other - the decision every schema-evolution argument is actually about.

How evolution breaks things

• A field removed without a default under backward compatibility is rejected at registration - which is the point; without the registry the same change fails silently at the consumer.
• Type changes (int to string, widening or narrowing) are almost never compatible; the registry rejects them and the producer is forced to add a new field or a new subject.
• Schema drift between environments - the schema registered in dev and the one in prod diverge because nobody promoted it through the same path. The registry makes the version history auditable; discipline makes it true.
• Subject naming collisions. The default strategy ties a subject to a topic; two unrelated record types in one topic need a different naming strategy (RecordNameStrategy or TopicRecordNameStrategy) or they share a compatibility contract they did not intend.

Karapace and the API

Confluent Schema Registry is the reference implementation; Karapace (Aiven’s open-source alternative) implements the same REST API and wire format, so the ecosystem of serializers and connectors works against either. The registry’s API compatibility is what makes the choice of implementation swappable - the contract is the wire format plus the REST surface, not the product.

Failure modes and diagnostics

• Registry unavailability. Serializers cache schemas but must reach the registry to register new ones or fetch unseen ids - a down registry blocks new-schema producers and cold-cache consumers. Run it highly available and treat it as infrastructure, not a sidecar.
• Compatibility rejection storms. A producer deploying an incompatible schema fails on registration - loud and correct, but it means the deployment cannot proceed until the schema is fixed or the mode is relaxed.
• Consumers reading without the registry. A consumer that deserialises by convention rather than by schema id works until a compatible-but-changed schema lands - then the failure is a wrong parse, not a clean error.
• Deleted subjects that were still referenced. DELETE /subjects/... removes the registration, not the data - records still carry the id and consumers can no longer resolve them. Soft-delete and reference counting exist for a reason.

What this is not for

A registry does not validate business meaning - it checks structural compatibility, not whether amount is still measured in pence. It is not a data catalogue: it does not tell you who produces or consumes a topic or what the field means, only what shape it has. And it is only worth operating where schemas actually change and consumers are independent - a single producer-consumer pair with a shared codebase gains less from it than a platform with many teams publishing to shared topics.