Let’s make this concrete with a small Quarkus application. An MCP weather tool returns data, an A2A summarizer turns it into a briefing, and an A2A translator translates that briefing. Apicurio Registry supplies the endpoint metadata. Application code supplies the sequence.

Two paths through the system

Metadata:   publishers -> Apicurio Registry -> application discovery

Execution:  application -> MCP weather -> A2A summary -> A2A translation

The Registry is not forwarding the weather result or choosing the next agent. The application asks for the required roles, resolves their endpoints and passes each actual result to the next step. This is deliberately a fixed workflow: its value is making the integration boundaries visible, not demonstrating autonomous planning.

The weather is fictional fixture data. The protocol calls, Registry requests and model invocations execute for real. That combination lets us inspect the plumbing without relying on a live weather provider.

Run the connected example

The companion repository wraps a pinned Quarkus LangChain4j sample. You need Git, Python 3.9+, Docker Compose, Java 25 to match the recorded environment, and a running host Ollama instance.

git clone --branch talk/evidence-2026-10-23 \
  https://github.com/carlesarnal/agent-discovery-demo.git
cd agent-discovery-demo
ollama pull qwen2.5:7b
python3 scripts/workflow.py prepare
bash scripts/paris-2026-demo.sh

Open http://localhost:11020/. The dashboard shows the briefing stages and the Registry versions observed before execution. In another terminal, run:

python3 scripts/workflow.py governance

The wrapper fetches commit 682928b714574e1ca8f0405c4319b36cd244dc0e from the sample associated with PR #2932. That PR was open when the example was checked. The pin is important: the sample builds its extensions from source, so it is not claiming that this workflow is available in every released extension version.

Remove a capability and inspect the result

The governance sequence starts with a successful briefing. It then attempts to publish a translator Agent Card with the translation skill removed.

Under the configured compatibility rule, Registry rejects the update. A useful test does more than assert an error response:

A compatible addition follows the rejection. It creates a registered version and the application runs the briefing again. Testing both paths avoids turning governance into a demonstration where every update is simply forbidden.

Fresh metadata needs correct publication

There is another failure mode before compatibility checking even begins: an agent restarts with a changed card but the publisher leaves the old card in Registry.

I addressed that in Quarkus LangChain4j PR #2897, merged on September 30, 2026. The change uses find-or-create version behavior, handles changed card content when the declared agent version is unchanged, and updates artifact labels. It includes a regression test against Registry.

Publication freshness and compatibility are separate responsibilities. A compatibility rule cannot inspect a change that was never submitted. Conversely, publishing current metadata does not establish that the remote implementation behaves correctly.

Where the guarantees stop

The successful rerun uses an unchanged running translator after a rejected metadata update. It does not prove that Registry prevents someone from deploying incompatible translator code. Connecting those controls requires deployment and behavioral tests owned by the application team.

Similarly, the versions displayed by the dashboard are observations, not transactional deployment pins. The local example has no deprecation-routing gate and no production authentication setup. The MCP registration is explicit in the runner, while A2A publication uses the extension’s publisher.

When a registry earns its place

If three services are released together by one team, configured URLs may be enough. A Registry becomes more useful when teams publish independently and callers need a shared place to inspect capabilities and their evolution.

The implementation includes the interactive flow and separate test and smoke commands. Stop the interactive runner before starting the smoke test; it creates its own services. The goal is not to remove application responsibility. It is to make the responsibility for discovery, publication and execution explicit enough to test.

Between those two moments was work that is easy to leave out of an acceptance announcement: defining the project boundary, explaining its relationship to neighboring work, and assigning responsibility for what happens after the vote. This post uses the public application record to make that work useful to other maintainers.

An application is a description that other people must evaluate

Maintainers already know what their software does. People reviewing a contribution to a foundation do not share all of that context. They need to understand which repositories are involved, how decisions are made and why the project fits the ecosystem.

The earlier application contains a good example. A question about using Apicurio with the Kubernetes API required clarification: deploying Registry through an operator, managing Registry artifacts through Kubernetes, and using Registry to describe another API are different integrations.

The lesson is not that every application must implement all of them. It is that an answer like “we integrate with Kubernetes” is too broad to tell a reader which capability actually exists.

Defining the boundary came before continuing

My January 23, 2024 comment recorded the decision to defer over branding questions and engagement with the Discovery API/xRegistry effort. It was a decision to clarify the contribution, not evidence that the software had stopped being useful.

The renewed application explicitly identifies the Registry repository in scope and describes the consolidation of Studio functionality. It also explains the intended relationship between a registry implementation and the xRegistry specification.

That distinction matters in an application just as it does in technical documentation. Implementing a specification, intending to implement it and participating in its community are related but different claims. A reviewer should be able to tell which one the project is making and what evidence supports it.

The vote and onboarding are separate milestones

The renewed application’s vote passed on March 12, 2026. A subsequent message required the contribution agreement before official onboarding could begin. The onboarding issue then listed work spanning project and foundation responsibilities.

Examples include licensing and trademark arrangements, repository hosting, governance and security documentation, community channels and project services. These are not all engineering tasks, and they are not all tasks a maintainer can complete alone.

It is tempting to compress that sequence into “the project was accepted.” For planning, the distinction is more useful:

Define the contribution
        ↓
Substantiate the application
        ↓
Review and vote
        ↓
Agreements and onboarding
        ↓
Ongoing project stewardship

The public record establishes that these steps occurred. It does not establish that one feature caused acceptance, or that I personally completed every onboarding task. Those would be different claims requiring different evidence.

Turn broad claims into a worksheet

The companion worksheet uses a simple structure:

For example, “we have open governance” should lead to a public decision process, named responsibilities and a path for an outside contributor to participate. “We fit this standard” should identify implemented behavior or explicitly say that alignment is planned.

Evidence should also be maintainable. A link to a changing main branch may be sufficient for exploration, but a historical account benefits from a dated reference. Record what the evidence established at the time instead of treating the current state of a repository as a perfect reconstruction of the past.

Carry ownership beyond acceptance

Foundation participation does not remove the daily work of reviewing changes, maintaining releases or responding to users. An application can reveal gaps that remain relevant long after admission: unclear ownership, contribution instructions that assume internal knowledge, or security documentation nobody has responsibility to update.

Use the same worksheet after the milestone. Which promises require continuing work? Which documents need periodic review? Which services now have a foundation contact as well as a project owner?

There is no universal formula here. Requirements evolve, and another project’s situation may differ from Apicurio’s. Check the current CNCF project process before applying. The reusable practice is to connect each claim to something another person can inspect and each gap to someone who can act on it.

Share the decisions, not only the announcement

The acceptance announcement records an important milestone for Apicurio Registry. The earlier deferral and onboarding record provide a different kind of value: they show the questions a maintainer team needed to resolve.

If you are preparing your own project, begin there. Define the contribution, distinguish current capability from intent, and make the remaining work visible. That produces a clearer application—and a clearer project for contributors and users, regardless of the outcome of a foundation review.

In this post, we’ll connect Keycloak, Kafka’s Strimzi OAuth libraries, and Apicurio Registry, then test the boundaries. We’ll use successful requests as controls, deliberately deny operations, and inspect the resulting state. Along the way, we’ll see why acquiring a token is not enough to prove that authentication is enabled.

Two requests, two decisions

A Registry-aware Kafka producer has two separate connections to consider:

                      Keycloak
                      /      \
             access token    access token
                    /          \
          Schema lookup         Kafka produce
                 |                    |
        Apicurio Registry         Kafka broker
        token checks + RBAC       token checks + ACLs

Registry evaluates access to groups, artifacts and versions. Kafka evaluates operations on topics, consumer groups and other broker resources. Sharing a realm does not combine these permission models, and it does not make an access token appropriate for every service.

Our example uses two service accounts, publisher and reader. Registry maps their realm roles to sr-admin and sr-readonly. Kafka separately grants topic write access to service-account-publisher. The administrator role is convenient for preparing the local example; a production schema publisher should receive only the permissions it actually needs.

Run the example

You need Python 3 and Docker with Compose. The example starts Keycloak 26.0, Registry 3.3.3, and Kafka 3.9.0 from the Strimzi 0.45.0 image.

git clone --branch talk/evidence-2026-10-23 \
  https://github.com/carlesarnal/keycloak-strimzi-apicurio-security.git
cd keycloak-strimzi-apicurio-security
python3 verify.py

The runner starts its own Compose project, performs the checks and removes the project’s containers afterward. Its fixed credentials, plaintext connections and internal administrative broker listener are for this disposable local environment. The administrative listener is not exposed as a host port; it must not become an alternative route for production clients to bypass OAuth.

Enable the Registry OIDC tenant

The important Registry settings in this example are:

environment:
  QUARKUS_OIDC_TENANT_ENABLED: "true"
  QUARKUS_OIDC_AUTH_SERVER_URL: http://keycloak:8080/realms/lab
  QUARKUS_OIDC_CLIENT_ID: registry
  QUARKUS_OIDC_TOKEN_ISSUER: http://localhost:18230/realms/lab
  QUARKUS_OIDC_TOKEN_AUDIENCE: registry
  APICURIO_AUTH_ROLE_BASED_AUTHORIZATION: "true"

The addresses differ for a reason. Registry discovers keys through the Compose-network address, while the host-side client obtains its token through localhost:18230. The issuer check must match the issuer in that token. The Keycloak clients also have an audience mapper that includes registry in their access tokens.

An earlier configuration used APICURIO_AUTH_ENABLED. In this Registry 3.3.3 setup, that did not enable the OIDC tenant: a request carrying an invalid token still returned 200. Setting QUARKUS_OIDC_TENANT_ENABLED corrected the configuration, and the same negative check returned 401.

That is why the test begins with an invalid credential, rather than simply demonstrating that a valid credential works. A successful authenticated-looking request can also succeed when authentication is disabled.

Verify read access without granting mutation access

After the publisher creates an artifact, the reader fetches its version content. The reader then attempts to create another version.

The last two checks are important. They establish the resource outcome, not just the response code: the accepted version remains unchanged and the denied version does not exist.

These checks also separate three diagnoses. A 401 points toward authentication. A 403 with a valid identity points toward authorization. A successful lookup followed by a denied mutation shows that the resource and connection are working while the operation is forbidden.

Grant Kafka access independently

The broker validates tokens through Strimzi’s OAuth callbacks and maps preferred_username to the Kafka principal. The example grants these operations:

kafka-acls.sh --bootstrap-server kafka:9092 \
  --add --allow-principal User:service-account-publisher \
  --operation Write --operation Describe --topic orders

The runner executes this command inside the disposable broker container using the internal administration listener. Application clients use the OAuth listener instead.

The publisher’s record arrives in orders. The reader obtains a valid token but receives TopicAuthorizationException when attempting to produce. A broker read then confirms that only the publisher’s record is present. A console producer can log an asynchronous send failure without making its process exit status a sufficient assertion, so checking the actual record matters.

Apply the diagnostic method to your deployment

When an application fails to access a Kafka platform, follow the specific connection before changing permissions:

1. Identify whether the failure came from the broker or the serializer’s Registry request.
2. Check the intended issuer, audience and mapped principal at that service.
3. Compare the failing operation with a successful control using the same path.
4. Inspect the smallest relevant permission: a Kafka ACL and a Registry role are different controls.
5. Verify the resulting state, especially after a denied write.

The Compose example exercises the real token and authorization machinery. Kubernetes networking, TLS, token renewal and signing-key rotation add further deployment concerns; they do not change the need to test these boundaries separately.

The complete configuration and assertions are in the example repository. For a different enforcement model, our earlier Envoy and OPA article moves authentication and authorization to a proxy. In either design, the useful question is the same: which component made this decision, and what evidence shows that it enforced it?

I contributed Avro and Protobuf support to the Kroxylicious Record Validation filter. In this article, we’ll use its JSON Schema support with Apicurio Registry to examine that failure contract. We’ll reject invalid records, inspect a mixed batch, bypass the proxy, stop Registry and recover the write path.

Where validation happens

Producer ──produce request──> Kroxylicious ──accepted records──> Kafka
                                   |
                                   └──schema lookup──> Apicurio Registry

The producer still speaks the Kafka protocol. Kroxylicious inspects record values before forwarding them, and Registry supplies the schema identified by the proxy configuration.

This example uses Kroxylicious 0.24.0, Registry 3.3.3 and Kafka 3.9.1. The Record Validation filter is a Preview feature in the referenced implementation; use the versioned behavior shown here rather than assuming identical error or caching behavior across releases.

Try the complete sequence

You need Python 3 and Docker Compose:

git clone --branch talk/evidence-2026-10-23 \
  https://github.com/carlesarnal/kroxylicious-schema-validation.git
cd kroxylicious-schema-validation
python3 talk/demo.py

The runner starts a dedicated Compose project, registers the schema, generates the proxy configuration and runs six cases. It reads the broker after the important writes and removes its containers when finished. Producer output is saved under talk/target/producer-results.json.

Register a deliberately small contract

The order schema requires an identifier and a nonnegative amount:

{
  "type": "object",
  "required": ["orderId", "amount"],
  "properties": {
    "orderId": {"type": "string"},
    "amount": {"type": "number", "minimum": 0}
  }
}

Registry returns a contentId when this artifact is created. The runner substitutes that value into the filter configuration:

filterDefinitions:
  - name: order-validation
    type: RecordValidation
    config:
      rules:
        - topicNames: [orders]
          valueRule:
            schemaValidationConfig:
              apicurioRegistryUrl: http://registry:8080
              apicurioId: __ID__
              schemaType: JSON_SCHEMA
defaultFilters: [order-validation]

__ID__ is a generation placeholder, not a value to send to the proxy. This example pins the filter to registered content; it is not a demonstration of automatically selecting the latest schema version.

One bad record can affect good records

A valid order reaches Kafka. An order with a negative amount produces InvalidRecordException and is absent from the topic.

Next, the runner submits a valid order and an invalid order in a batch, using a single-partition topic and producer settings that allow them to batch together. In the tested setup, both records are rejected. Reading Kafka confirms that the valid member of the failed batch did not arrive either.

This matters for application error handling. “Retry the valid record” may be the right next step for this known rejected batch, but it is not a complete general recovery algorithm. Producers must account for ordering requirements, ambiguous outcomes and duplicate handling. A broker timeout, for example, must not automatically be interpreted as the same definite rejection demonstrated by this test.

A proxy only protects its path

The next case sends an invalid record directly to Kafka’s internal broker endpoint. Kafka stores it because that request did not pass through the validation filter.

There is no contradiction: the broker has not become schema-aware merely because a proxy exists. If proxy validation is a required control, broker access and network policy must enforce the path. Keep client-side validation too when possible; it can give the application faster, more contextual feedback before a network call.

A Registry outage looks different from invalid input

Stopping Registry creates a more subtle case. We send a valid order through the proxy and receive an InvalidRecordException containing:

context=SCHEMA_RESOLUTION_ERROR

The record is rejected because its schema cannot be resolved, not because its amount violates the schema. The exception class alone therefore cannot decide whether an application should correct its input or investigate an infrastructure dependency.

Recovery means restoring the schema, too

The recovery test initially failed even after the Registry container came back. Our local Registry uses ephemeral storage: restarting the process restored HTTP availability but lost the schema. The proxy then reported ContentNotFoundException for its configured content ID.

The corrected laboratory sequence re-registers the schema and checks that the restored contentId matches the proxy’s configuration before sending a new record. That record reaches Kafka. In a durable deployment, the corresponding responsibility is preserving and restoring Registry data, not hoping that a healthy process has the same content as before.

This recovery is intentionally narrow. It demonstrates restoring the dependency and successfully sending a new record. It does not claim a general exactly-once resend strategy.

Design the failure contract before enabling enforcement

The proxy adds useful protection, particularly when producers cannot all be changed together. Its operational cost is that schema resolution, batching and routing now participate in write behavior.

Before enabling it, decide how producers distinguish constraint errors from resolution errors, how failed batches are handled, and how direct access is controlled. The runnable example turns those questions into assertions against actual broker contents. That is the useful test of an enforcement point: not only whether it reports an error, but what it allows to be stored.

Maintaining an open-source project involves more than deciding whether a patch is correct. Contributors need to know who owns the next step, what evidence is missing and what happens when that person is unavailable. The project’s limited review capacity needs to be visible too.

Apicurio Registry’s published PR lifecycle provides a concrete example of making those hand-offs explicit. Let’s look at its state model and one public review, then extract the parts another project can use without adopting the same automation.

Inactivity needs an owner

The documented lifecycle distinguishes waiting-on-author from waiting-on-maintainer. That difference affects escalation: a PR blocked only on a maintainer does not become stale and get automatically closed for contributor inactivity. Instead, it can receive a review-overdue signal.

That policy asks a better question than “when was this PR last updated?” Bots and CI can update a PR without a human moving the review forward. Conversely, an author may have nothing useful to add while waiting for an assigned reviewer.

The documented process also has an explicit trade-off: if both the author and maintainer are blocked, the contributor timer takes precedence. An ownership-aware process still needs decisions about ambiguous states; it does not make them disappear.

Separate the hand-offs

Here is a simplified view of the responsibilities:

The table is not a replacement for the actual repository rules. It is a way to inspect whether every state has a meaningful next action.

For instance, a suggested reviewer is not proof that the person has accepted ownership. A successful quick check is not a successful full verification suite. “Approved” is not necessarily “ready to merge now” if required checks are still running or a newer commit invalidated the review.

A real approval-to-verification sequence

PR #10409 fixed a pagination test that could include agent records created by other tests. The change scoped fixtures with a unique label and checked that the combined pages contained the expected agents, rather than assuming an ordering not guaranteed by the API.

The public record shows a reviewer suggestion, my approval on October 7, 2026, and bot comments distinguishing the passed quick check from the remaining full-suite gate. The PR merged later that day.

This is useful because each transition is inspectable. It does not prove that automation reduced review time or increased contributor retention. It shows how the contributor and maintainer can tell which decision or check remains outstanding.

It also illustrates why source changes and workflow state need different explanations. A contributor should understand the technical fix from the PR description and the next operational step from the lifecycle feedback, without reverse-engineering several CI dashboards.

Assignment is a policy, not just a notification

The documented reviewer-assignment approach considers ownership, interest and load. These signals represent different things: experience with the changed code, willingness to review an area and recent assignment pressure.

None is a perfect measure of availability. A specialist may appear lightly loaded because they cover fewer areas, not because they have spare capacity. A person who wrote much of the code may become a bottleneck if ownership dominates every assignment.

This is why an assignment system needs an explanation and a manual correction path. Contributors need a person to act; maintainers need to understand why that person was selected and how to change the policy when its assumptions stop fitting the team.

Start with the convention before the bot

You can use the hand-off worksheet without deploying any automation. Pick one delayed contribution and answer:

1. Who owns the next action?
2. What specific evidence or decision is required?
3. How can the contributor see that state?
4. Who can take over if the owner is unavailable?
5. What event makes the current status obsolete?

If these answers are unclear, automating a reminder will not clarify them. First agree on a small convention; then automate the repetitive parts that the team can realistically maintain.

Automation itself becomes project code. Labels can be stale, commands can race with new commits and a workflow can report the wrong state. Keep the authoritative checks tied to the relevant commit and expose enough context for people to recover from a mistaken automated message.

Measure what the process actually changes

Review waiting time and repeat participation are useful signals to investigate. They are not interchangeable with causality. A change in contribution volume, release timing or team availability can affect those numbers independently of a new lifecycle policy.

Start with a concrete outcome: fewer PRs where nobody knows the owner, fewer author-directed reminders while a maintainer is blocking progress, or a clearer path from approval to required verification. Then inspect whether the policy produces that outcome.

The goal is not a perfectly automated contribution pipeline. It is a process people can understand, correct and sustain. Making the next action explicit is a small improvement with value on both sides of the review.

Consider a customer record that changes twice during a rebuild, and another that is deleted. A successful bulk load can still leave the replacement index with an old value or a resurrected document. We need a baseline, the subsequent changes, an ordering rule and a controlled cutover.

The example below makes those requirements visible using real PostgreSQL, Debezium, Kafka and Elasticsearch services. It complements schema-governance work: Registry can describe the records, but schema validity cannot prove that a derived view is complete.

Keep the first recovery experiment small

Our lab uses PostgreSQL 16.4, Debezium 3.0.0.Final, Kafka 3.9.1 and Elasticsearch 8.17.0. It captures one table into a single-partition topic, retaining its complete history. A Python runner reads actual Kafka events and applies them to Elasticsearch in order.

PostgreSQL --WAL--> Debezium --events--> Kafka partition 0
                                             |
                                      ordered replay
                                             |
                              customers-old / customers-new
                                             |
                                      customers alias

This is an explicit replay implementation, not an Elasticsearch sink connector configured behind the scenes. The small scope lets the test assert exact contents at each stage.

Run it

Install Python 3 and Docker Compose, then use the maintained branch:

git clone --branch talk/evidence-2026-10-23 \
  https://github.com/carlesarnal/debezium-ocp-etc-demo.git
cd debezium-ocp-etc-demo
python3 recovery.py

The runner starts a dedicated Compose project, executes the recovery sequence, writes results.json and removes its containers and volumes. The services use local, disposable configuration; do not reuse these credentials or exposed development settings for production.

Establish the baseline

The source table begins with two rows:

CREATE TABLE customers(id integer PRIMARY KEY, name text);
ALTER TABLE customers REPLICA IDENTITY FULL;
INSERT INTO customers VALUES (1, 'before'), (2, 'delete-me');

Debezium snapshots them as two r events. The runner verifies that both arrived before it proceeds. That precondition matters: a recovery test that starts from an empty stream can pass while exercising almost none of the intended behavior.

Both rows are applied to customers-old, and the customers read alias initially points there. A separate customers-new index is created for the rebuild.

Change the source while the replacement loads

After loading only the first baseline event into the replacement, the runner performs these writes:

UPDATE customers SET name = 'middle' WHERE id = 1;
UPDATE customers SET name = 'after' WHERE id = 1;
DELETE FROM customers WHERE id = 2;
INSERT INTO customers VALUES (3, 'new');

It finishes loading the baseline and waits for all four changes. The captured operation sequence is now:

r, r, u, u, d, c

Record identity is the table’s primary key. Inserts and updates use Elasticsearch document PUT; deletes use document DELETE. Replaying the changes in partition order yields:

{"1": "after", "3": "new"}

The deleted key must be absent. A row count alone would not establish correctness: two wrong documents could still produce the expected count.

Replay again, then switch the alias

The runner applies the complete retained history again and checks the same final contents. This demonstrates convergence for complete, ordered replay in this fixture. It does not show that arbitrary out-of-order updates are safe: an older PUT after a newer one could overwrite it without an additional version guard.

Once the replacement passes its content checks, the alias changes in one Elasticsearch request:

{
  "actions": [
    {"remove": {"index": "customers-old", "alias": "customers"}},
    {"add": {"index": "customers-new", "alias": "customers"}}
  ]
}

The test queries the alias rather than just the physical index. It then updates customer 3 again in PostgreSQL, captures the seventh event and applies it to the replacement. The alias must now return after-cutover for that customer. Cutover is not finished if later writes still go only to the old index.

Where a production recovery plan needs more detail

The complete-history assumption is central. If required events have expired, replay cannot invent the missing baseline. A new snapshot must be coordinated with subsequent changes so neither gaps nor incorrectly ordered updates appear.

For multiple partitions, define progress per partition and the consistency the view requires. Do not compare unrelated offsets as one global clock. Also decide how deletion information survives retention or compaction, how replay resumes after a crash, and whether reads can tolerate remaining lag at cutover.

The fixture does not demonstrate these more complex paths, and it does not make the source and index instantaneously equal while writes continue. It provides a concrete starting point for testing them: known source changes, observed events, exact document assertions and a read-path switch.

Make “disposable” an executable claim

A useful rebuild runbook should answer four questions: where the baseline comes from, how later changes are ordered, what proves the replacement is ready, and where new changes go afterward.

The implementation and recorded result turn those questions into a seven-event integration test. The same reasoning applies to other derived views: recovery is a behavior to exercise, not a property obtained simply by calling a store a cache or an index.

I encountered this while assembling a PostgreSQL-to-Iceberg pipeline. The sink’s control consumer could not maintain its group membership, and its coordinator reported commits to zero tables. The task’s lifecycle status was accurate: it was running. It simply was not making data available to the query engine.

This article follows the evidence needed to distinguish those two states. Apicurio Registry is part of the pipeline as the Avro schema service; the Iceberg catalog has a separate responsibility.

Map the data path before reading every log

PostgreSQL -> Debezium -> Kafka -> Iceberg sink -> object storage
                 |                  |                  |
                 +-- Avro schemas --+             data files
                          |                            |
                   Apicurio Registry         Iceberg REST catalog
                                                     |
                                                   Trino

Debezium emits changes, Kafka retains them, the sink writes data and commits table metadata, and Trino queries a committed table view. A success at one boundary is not proof that the next boundary succeeded.

The maintained Kubernetes lab uses Registry 3.3.3, the Apache Iceberg Kafka Connect sink 1.11.0 and a separate Iceberg REST catalog fixture. An older version of the repository described Registry as the catalog; that is not the architecture reproduced by this lab.

Follow one key

Start with a known source change rather than a cluster-wide dashboard. The companion progress-ledger.txt organizes the investigation:

An offset is meaningful in its partition. Comparing the largest offset across unrelated partitions does not establish a global progress boundary.

Registry lookup is another boundary to inspect when deserialization fails. A readable Kafka record does not imply the sink has the schema or references it needs. Conversely, successful schema resolution does not imply that an Iceberg commit completed.

The control consumer was the useful clue

The failed setup logged UNKNOWN_MEMBER_ID during the sink control consumer’s classic group-protocol interaction. Commit attempts timed out with committed to 0 table(s) even while the task reported RUNNING.

In this setup, changing the sink’s Kafka group protocol restored commits:

iceberg.kafka.group.protocol: consumer

That is an observed recovery, not a universal repair instruction. The underlying interaction was not isolated far enough to claim a root cause for every occurrence of that error. The portable lesson is where to look: the sink has coordination and commit work beyond consuming the data topic.

A commit interval is not a freshness guarantee

Two recorded local runs made an inserted row queryable after 33 seconds and 42 seconds, despite this setting:

iceberg.control.commit.interval-ms: 10000

The measurements include polling and query overhead. They do not isolate the time spent in capture, consumption, coordination, commit or query visibility. Treating the ten-second setting as an end-to-end guarantee would hide precisely the delays we need to understand.

Likewise, initial snapshot ingestion includes table creation and startup work. Its timing is not directly comparable with a single change flowing through a warmed pipeline. Keep startup and steady-state observations separate before drawing conclusions about freshness.

The table is a changelog, not automatically a mirror

The example sink appends flattened change events and a _cdc structure. After an update, the table contains both the original insert and the update event. Seeing the update in a snapshot is different from having a query that returns only the current row.

The example uses a latest-event-per-key query:

SELECT id, first_name, last_name, phone
FROM (
  SELECT *,
         row_number() OVER (
           PARTITION BY id ORDER BY _cdc.offset DESC
         ) AS rn
  FROM iceberg.inventory.customers
)
WHERE rn = 1 AND _cdc.op <> 'D';

The ordering expression must match the semantics of your connector’s position field and the events being compared. Do not generalize it into a global order across arbitrary partitions or sources. For larger workloads, an append-only changelog also raises questions about compaction, snapshot retention and whether to materialize current state rather than reconstruct it on every query.

Try the pipeline

The maintained repository branch contains the Kubernetes implementation under talk/lab/. Use a disposable cluster and review its README for the pinned component versions and local image-loading assumptions.

git clone --branch talk/evidence-2026-10-23 \
  https://github.com/carlesarnal/database-to-lakehouse-demo.git
cd database-to-lakehouse-demo/talk/lab
./connect-image/build.sh
./scripts/01-deploy.sh
./scripts/run-demo.sh
./scripts/99-teardown.sh

The teardown deletes the lab namespace. The recorded runs used minikube on macOS ARM64; the catalog fixture and single-node services are development infrastructure, not a production availability design.

Monitor the outcome users depend on

Process health, ingestion progress and query freshness answer different questions. Keep all three, but do not let the first stand in for the others.

When the next pipeline stalls, select one key and identify the last boundary with convincing evidence. That gives you a smaller investigation than “Kafka Connect is green but analytics is broken,” and a much better starting point for an alert that reflects what users actually experience.

My earlier classification application connected Kafka, Spark and a Quarkus results service. To investigate processing behavior without mixing every stage together, I built a smaller experiment that isolates Spark Structured Streaming. It uses timestamped file input, one small classifier and a deliberately visible admission limit.

The interesting result was not merely that slow processing creates backlog. Removing the injected delay improved prediction age, but did not clear the next bottleneck.

Define the measurement boundary

The experiment runs Spark 3.5.3 in local[2] inside a container limited to two CPUs and 3 GB of memory. Its model is a CountVectorizer and MultinomialNB pipeline trained on four explicit fixture texts:

TEXTS = [
    "database query schema",
    "football match goal",
    "database table index",
    "football team stadium"
]
LABELS = ["technology", "sport", "technology", "sport"]

This classifier supplies actual prediction work, but its training set is intentionally tiny. Correct labels for these fixtures demonstrate pipeline execution, not useful generalization or production model quality.

Each input carries an event ID and a new ingress timestamp. The measured age ends when the foreachBatch callback records the prediction. It includes Spark scheduling and collection plus the classifier and any injected delay; it does not include Kafka ingestion, Registry lookups or HTTP query visibility.

Run the comparison

git clone --branch talk/evidence-2026-10-23 \
  https://github.com/carlesarnal/reddit-realtime-classification.git
cd reddit-realtime-classification
docker build -t talk-stream-experiment:local experiment
docker run --rm --cpus 2 --memory 3g \
  -v "$PWD/experiment:/lab" talk-stream-experiment:local

The image contains Spark and scikit-learn 1.3.2. The runner writes per-event and per-batch observations to experiment/results.json. The current branch is the focused experiment; the earlier live-feed application remains in repository history.

Make admission visible

The source has a deliberately restrictive configuration:

stream = (
    spark.readStream.schema(schema)
    .option("maxFilesPerTrigger", 1)
    .json(str(source))
)

The query uses a one-second processing-time trigger. Each file contains one event. A warm-up event runs before measurement, then twelve events are offered in each case:

1. Baseline: a nominal 1.2 seconds between events, no injected delay.
2. Overload: a nominal 0.1 seconds between events, with 1.2 seconds of injected work per record.
3. Delay removed: the same nominal 0.1-second input interval, with the injected work removed.

The actual offered rate is measured, not assumed from sleep(). File creation, scheduling and system load make the real rate differ from the nominal interval.

Read the observations together

One local run produced:

Removing the delay improved age, but it did not make the workload sustainable. The query still admitted only one file per trigger, while files arrived much faster. The pending count at the end of input is a snapshot, not the final loss count: the runner subsequently drains all available input and asserts that all twelve IDs were processed.

These are single-run observations, not a performance benchmark with confidence intervals. The input rates differ somewhat, and the slow case uses an artificial delay rather than claiming that the real classifier takes 1.2 seconds. Those qualifications are essential to interpreting the comparison correctly.

Avoid measuring the wrong age

A replay dataset may carry old source timestamps. Reusing them as “ingress time” would make every prediction look old even if processing were immediate. Keep source-event age and time spent in the current pipeline as separate metrics.

The runner records:

event ID -> ingress -> batch processing start -> result recorded

It also records batch duration and event count. These fields let us inspect whether delay accumulated before processing or during the batch. Because frame.collect() brings records to the driver and the classifier runs there, this particular experiment does not measure distributed executor-side inference throughput.

Extend the experiment one boundary at a time

For a production diagnosis, repeat runs with representative input and resource limits. Then include the other stages: Kafka arrival, schema resolution if applicable, model completion, output publication and the application’s queryable result.

Changing a micro-batch interval, raising a source-admission cap and adding inference capacity solve different problems. Raising the cap can improve admission but create larger batches or exceed downstream capacity. More API pods may have no effect on any of those limits.

Schema governance and observability complement each other here. A Registry can help preserve the shape and meaning of exchanged data; progress measurements tell us whether that data arrives in time to be useful. Neither a valid schema nor a healthy endpoint establishes freshness.

The source and raw results make the example reproducible. Use the same discipline on your own pipeline: state the measurement boundary, change one controllable cause, and look for the next limiting stage before adding capacity.

It is not accuracy. Two models can confidently make the same mistake. This article explains how to use agreement without turning it into a guarantee, and how to keep model-quality monitoring separate from the processing measurements in the current streaming experiment.

The missing variable is the answer

Let A(x) and B(x) be two model predictions. Agreement asks whether:

A(x) == B(x)

Accuracy requires a label y(x):

A(x) == y(x)

The first expression contains no evidence about the second. Different architectures can reduce some shared failure patterns, but they can still share training data, label definitions, preprocessing errors or a systematic blind spot.

Here is a deliberately small example:

The models agree on two of three inputs, and both agreed answers are wrong. Treating agreement as a confidence score would prioritize the wrong cases for automatic acceptance.

Disagreement is still useful

Disagreement creates a queue for investigation. It can reveal ambiguous examples, a changed category distribution, a preprocessing bug or one model behaving differently after an update. The signal is a reason to inspect, not a diagnosis by itself.

Keep the model versions, preprocessing version and input identifier with the comparison. Otherwise a jump in disagreement may be difficult to distinguish from a deployment that mixed old and new predictions.

A cross-tabulation of two models’ labels is also not a ground-truth confusion matrix. It shows where the models disagree with each other. Use names that make that distinction visible on dashboards and in alerts.

Confidence scores need their own evaluation

One model reporting 0.95 while another reports 0.70 does not prove the first is more likely to be correct. Confidence scales can differ across architectures and training procedures. Calibration asks whether scores correspond to observed correctness on representative labeled examples.

Without that evaluation, rules such as “prefer the more confident model” simply replace one unverified assumption with another. Even calibrated scores can become unreliable when the input population changes.

An operational policy might instead send selected disagreements for review, use a conservative fallback for high-impact cases, or delay action until a label arrives. The appropriate policy depends on the cost of the error, not only the model scores.

Sample agreements as well as disagreements

If every reviewed example comes from the disagreement queue, shared mistakes remain invisible. Include a representative sample of agreements, and avoid reporting review-set accuracy as population accuracy when the sampling strategy is biased.

Useful dimensions include category, language, input length and source. A stable global agreement rate can conceal degradation in a small but important segment. Track coverage as well: missing predictions, rejected inputs and timeouts should not silently disappear from the denominator.

When delayed labels become available, connect them back to the predictions made at the time, including their versions. Re-running a newer model on the input answers a different question from evaluating the original decision.

Quality and freshness are separate axes

A correct prediction can arrive too late to be useful. A fast prediction can be wrong. Measuring one does not establish the other.

The maintained Spark experiment deliberately uses one tiny, fixed classifier. It records ingress and processing timestamps while varying input pacing and an injected processing delay. The fixture labels verify that the expected path executed; four training texts do not establish useful generalization.

That experiment avoids mixing model disagreement with queueing behavior. Its companion article explains the measured backlog and source-admission limit. The older dual-model application remains in repository history, but the current branch should not be described as running a production disagreement-monitoring platform.

A practical monitoring split

Schema checks can protect the structure of these records, and feature-input invariants can catch semantic changes before inference. Neither establishes that a classification is right. Use the data-contract example for that separate input boundary.

The strongest use of agreement is modest and practical: it helps decide where to look. Keep it beside labeled evaluation, coverage and freshness, and it becomes a useful operational signal without pretending to supply the ground truth you do not yet have.

The words alone do not decide. An MCP tool can invoke a sophisticated model, and an A2A agent can perform deterministic work. What matters is the contract presented to the caller: discovery, input and output, progress, failure and cancellation.

To compare those contracts, I built a small application that exposes exactly the same function through the official Python SDKs. Removing model variability lets us see the protocol behavior rather than debate two different generated summaries.

Hold the underlying work constant

Our summarizer is intentionally simple:

def summarize(text):
    if not text.strip():
        raise ValueError("text must not be blank")
    return " ".join(text.split()[:8])

Both interfaces receive:

One implementation can expose multiple interfaces without changing the underlying work.

Both must return:

One implementation can expose multiple interfaces without changing

This is not a language-quality test. It gives us a stable oracle for the transport and lifecycle comparison. Once those behaviors are understood, a real model can replace the fixture without changing the questions we ask of its interface.

Start both servers and run the comparison

The example pins A2A SDK 0.3.22, MCP SDK 1.26.0, Uvicorn 0.40.0 and HTTPX 0.28.1. Use Python 3.13 and uv to match the tested client environment:

git clone https://github.com/carlesarnal/a2a-mcp-comparison.git
cd a2a-mcp-comparison
uv venv .venv
uv pip install --python .venv/bin/python -r requirements.txt
.venv/bin/python integration.py
.venv/bin/python compare.py --require-observed

The integration runner starts loopback servers on ports 18201 and 18202, makes real HTTP calls, records observations and stops both processes. A2A advertises protocol 0.3.0; the MCP session negotiates 2025-11-25 in the recorded run.

Discovery exposes different application surfaces

The A2A server publishes an Agent Card with a summary skill. The test fetches /.well-known/agent-card.json and verifies that declaration before sending a message.

The MCP client initializes a session and calls list_tools(). Its summary tool exposes typed arguments, including text and an optional delay used to make progress observable.

Neither discovery response proves the function works. We still invoke the capability and compare its result with the expected eight-word string. This is also where Registry-backed discovery fits: a registry can help locate and govern metadata, but it does not replace the protocol’s invocation contract.

Failure is part of the interface

Send blank text and the shared function raises ValueError. The two adapters communicate that failure differently:

These are observations of the supplied implementations, not a complete feature comparison between the standards. In particular, the MCP example does not implement a durable task store, and the A2A example uses an in-memory task store.

Cancellation needs an observable boundary

For A2A, the runner starts delayed work in nonblocking mode, polls until the task is working and sends tasks/cancel. It then checks for a canceled state and no result artifact. The executor cooperatively observes a stop event.

For MCP, the client starts a delayed tool call and waits for progress before cancelling its pending operation. It observes client-side cancellation and proves that another tool call on the session still works.

Those are not equivalent guarantees. A canceled client operation does not, on its own, prove that server-side computation stopped. Neither example proves recovery after the server process dies. Keeping these boundaries explicit is more useful than declaring that both protocols “support cancellation” and leaving callers to discover what that means.

Choose from caller requirements

Before choosing an interface, write down what the caller needs:

• Is a typed request/result tool sufficient, or does the application need a task it can address later?
• How will progress reach the caller?
• What should a timeout or cancellation mean for work already in progress?
• Which implementation features are advertised, and which have actually been exercised?
• What state must survive a process restart?

Exposing both interfaces can be reasonable, but it adds two contracts to maintain. Shared business logic does not remove adapter-specific errors, authentication or lifecycle tests.

The repository keeps the implementation, observations.json and comparison table together. The useful output is not a winner. It is a vocabulary for choosing an interface whose observable behavior matches the application you are building.