Skip to main content

Technical Advisory 10016

Date​

Versions:1

  • v2.65.x: > v2.65.9

Date: 2025-05-14

Last updated: 2025-05-14

Description​

Background​

Zitadel uses a eventstore table as main source of truth for state changes. Projections are tables which provide alternative views of state, which are built using events. In order to know which events are reduced into projections, we use a position column in the eventstore and a dedicated table which records the current state.

Problem​

Zitadel prior to the listed version had a precision bug. The position column uses a fixed-point numeric type. In Zitadel's Go code we used a float64. In certain cases we noticed a precision loss when Zitadel updated the current_states table.

Impact​

During a past attempt to fix this, we got reports of failing projections inside Zitadel. Because the precision became exact certain compare operations like equal, less then, etc would now return different results. This was because the values in current_states would already have lost precision from a broken version. This might happen to some deployments or projections: there is only a small probability.

We are releasing the fix again and your system might get affected.

  • Original issue: 8671
  • Follow-up issue: 8863

Mitigation​

When after deploying a fixed version and only when experiencing problems described by issue 8863, the following queries can be executed to fix current_state rows which have "broken" values. We recommend doing this in a transaction in order to double-check the affected rows, before committing the update.

begin;

with
  broken as (
    select
      s.projection_name,
      s.instance_id,
      s.aggregate_id,
      s.aggregate_type,
      s.sequence,
      s."position" as old_position,
      e."position" as new_position
    from
      projections.current_states s
      join eventstore.events2 e on s.instance_id = e.instance_id
      and s.aggregate_id = e.aggregate_id
      and s.aggregate_type = e.aggregate_type
      and s.sequence = e.sequence
      and s."position" != e."position"
    where
      s."position" != 0
      and projection_name != 'projections.execution_handler'
  ),fixed as (
    update projections.current_states s
    set
      "position" = b.new_position
    from
      broken b
    where
      s.instance_id = b.instance_id
      and s.projection_name = b.projection_name
      and s.aggregate_id = b.aggregate_id
      and s.aggregate_type = b.aggregate_type
      and s.sequence = b.sequence
  )
select
  b.projection_name,
  b.instance_id,
  b.aggregate_id,
  b.aggregate_type,
  b.sequence,
  b.old_position,
  b.new_position
from
  broken b;

If the output from the above looks reasonable, for example not a huge difference between old_position and new_position, commit the transaction:

commit;

When there are no rows returned, your system was not affected by precision loss.

When there's unexpected output, use rollback; instead.

Footnotes​

  1. The mentioned fix is being rolled out gradually on multiple patch releases of Zitadel. This advisory will be updated as we release these versions. ↩