Delegated access management

Hosted Login

Multifactor Authentication

Passwordless Authentication

Audit Trail

Identity Brokering

Platform

Service Accounts

Features

Easy integration

B2B (Business to Business)

Self Service

Existing identities

Secure authentication

Use Cases

Learn how to install, setup, and use ZITADEL.

Documentation

ZITADEL Cloud

Self-hosting

Trust center

Examples and Guides

Github Repository

Develop

Get Started

Contribute

Roadmap

Changelog

About Us

Jobs

Contact

Company

Read the blog

Sign up

Login

Pricing

Blog


## Description

We were busy since our last Into the CAOS and want to share some updates around ZITADEL and what is planned in the near future.

We will also talk about new features such as Metadata, Actions, new policies as well as what did not go so well.

<br />
<div className="my-4">
  <YouTubeExtended
    videoId={"RZJkNX7s7Zk"}
    containerClassName={"youtube-container"}
    stopsTitle="Topics covered"
    stopList={[
      { name: "Highlights from the past weeks", time: "1:50" },
      { name: "Metadata", time: "5:20" },
      { name: "Handling External Identity Providers", time: "7:06" },
      { name: "Actions", time: "9:00" },
      { name: "Lockout policy", time: "17:00" },
      { name: "Query Side", time: "21:06" },
      { name: "Question to Lockout policy", time: "25:21" },
      { name: "Native Backup", time: "28:19" },
      { name: "OpenAPI/Swagger", time: "30:10" },
      { name: "Issues and learnings", time: "33:29" },
      { name: "New languages", time: "39:38" },
      {
        name: "How to use ZITADEL for Product vs. Organizations",
        time: "42:36",
      },
      { name: "Roadmap", time: "54:31" },
      { name: "Penetration test", time: "58:40" },
    ]}
  ></YouTubeExtended>
</div>

## References

- [Swisscom Cybersecurity Startup Challenge](https://www.swisscom.ch/en/magazine/start-up/startup-challenge-security/)
- [DINAcon Awards](https://awards.dinacon.ch/gewinner2021/)
- [API Documentation](https://zitadel.com/docs/apis/introduction/)
- [Internationalization Documentation](https://zitadel.com/docs/guides/manage/customize/texts#internationalization)
- [Smartive - Built with ZITADEL](https://zitadel.com/blog/success-story-smartive-bin)
- [Pentest results](https://zitadel.com/blog/pentest-results-h1-2021)

## Discuss features

- [Discussion: Metadata on Orgs](https://github.com/zitadel/zitadel/discussions/2454)
- [Discussion: Project settings](https://github.com/zitadel/zitadel/discussions/2263)


We were busy since our last Into the CAOS and want to share some updates around ZITADEL and what is planned in the near future.

Into the CAOS | No 2.


During the first half of 2021 an external specialist conducted a security analysis of ZITADEL. We want to show you the results, what we already did and what we’re planning to do in the future.

## TL;DR

The overall result was rated as good with only one high issue, which we already mitigated partially. Other issues were minor but nonetheless important for us and were fixed within a few days. Others still need more thinking from our side to find a solution that not only solves the immediate risk, but also gives the users a real benefit. Until then a lot of the findings can be mitigated from the users themself by using either Passwordless (WebAuthN) or at least by protecting their accounts with a second factor (2FA).

## Findings

In the following section I will give you a short summary and comment on the highest ranked issues. The number indicates the external assessment according to the Common Vulnerability Scoring System (CVSS).
[You can find the full report here.](/blog/2021-11-11-pentest-results-h1-2021-01.pdf)

### 8.1 Data not Encrypted at Rest (CVSS 8.2 / High)

> “The ZITADEL Console web application does not encrypt data at rest. An attacker with access to the database, can access the stored secret and personal identification information data.”

Although ZITADEL does not encrypt its database, the storage (hard disks) is encrypted. Furthermore, ZITADEL either hashes (bcrypt) or encrypts (AES) certain events for example passwords, OTP Secrets, private keys or temporary codes.

### 8.2 Failed Login Attempts Not Displayed (6.5 / Medium)

> “The ZITADEL Console web application does not display failed login attempts after a successful authentication. An attacker may profit of this missing control, since a user will not know, that he was subject to an attack.”

We certainly lack a corresponding display and we’re planning to have such in the near future. For the moment a user is able to see these and all other events that have occurred on his user on the “Personal Information” page in the “Activity” section:

<img src="/blog/2021-11-11-pentest-results-h1-2021-02.png" width="60%" alt="" />

To mitigate the risk of an attacker attempting to log into your account, we highly recommend using Passwordless (WebAuthN) or at least protect your account with a second factor (2FA) for example Universal Second Factor (U2F) or One Time Password (OTP). ZITADEL allows managers to enforce the use of multi factor or passwordless authentication.

### 8.3, 8.4, 8.6 Missing Protections for Username / Password Authentication

I would like to take a look at the three points together, because the focus is on protecting authentication using usernames and passwords. To say it in advance, these can be mitigated considerably by using WebAuthN or at least a 2FA mechanism:

- 8.3 Missing Protection Against Online Guessing (6.5 / Medium):
  > “The ZITADEL Console web application does not protect user accounts with an application layer mitigation mechanism against automated brute-force attacks. An attacker with access to the login page can perform an online brute-force attack to guess a user’s password.”
- 8.4 Consider Restriction of Common or Compromised Passwords (6.5 / Medium):
  > “The ZITADEL Console web application does not offer the option for IAM Administrators to restrict the usage of common and compromised passwords. An attacker can be successful using a wordlist based brute force attack to takeover user accounts.”
- 8.6 Username Enumeration Possible (4.3 / Medium):
  > “The ZITADEL Console web application login allows enumeration of usernames, due to a warning that is showed after entering a non-existing username. An attacker with access to the login page can perform an online brute-force attack to enumerate existing usernames.”

To mitigate against brute-force attacks at least on a network layer, we use an IP-based rate limiting for our cloud service and dedicated instances. While we also provide a possibility to create a lockout policy, we do not recommend using it, as it also enables attackers to Denial-of-Service a user. Therefore a reasonable solution would be to use Captchas and prompt users to solve it when a user fails to enter a correct password multiple times.

When using a weak password an attacker's chance to take over an account is even higher. Giving organisation administrators the possibility to restrict certain passwords or make users aware that the password they use, might be weak or compromised will be an enhancement we plan to implement in the future.

We’ve discussed a lot whether we should show a warning that the entered username does exist or not. But the fact that a user can authenticate using external identity providers (e.g. Google) somewhat made it obsolete, because when the user enters their username, we might have to redirect them to their identity provider, which will disclose the fact that the user exists. Moreover, a malicious user who already has access to ZITADEL might gain access to this information anyway (e.g. through the client application).

### 8.5 IAM Policies Not Enforced (5.3 / Medium)

> “The ZITADEL Console web application fails to disable user registration and password
> authentication on the server-side. An attacker who knows the technical details, on how to
> register a user, can register a user without access to any projects or organization
> information.“

A registration of a user, which would not have been allowed is certainly a problem and we fixed it. However, there are possible measures and facts that mitigate this issue. First of all a successful registration only allows a user to authenticate, which he might already be able to do so in another organisation, but no authorization to any project is granted by default. So if applications do their supposed authorization checks, they will not have a problem. As the user will not get any manager role within ZITADEL, he will as well not gain access to any information of other users of his organization.

### 8.7 Verbose Error Messages (4.3 / Medium)

> “The ZITADEL Console web application shows verbose error messages in the HTTP Header Grpc-Message. This information can be helpful for an attacker to mount further attack steps.”

Due to the fact that ZITADEL is OpenSource, a lot of this information can be obtained anyway. But we definitely see room for improvement here, especially from a usability aspect and we will continuously improve this.

### 8.8 Consider Usage of Refresh Token (3.3 / Low)

During the security analysis, refresh tokens could not be tested because we were still implementing them. We released this feature the following week (see [v1.8.0](https://github.com/zitadel/zitadel/releases/tag/v1.8.0))

### 8.9 Logout Does Not Terminate Session (3.3 / Low)

Although this was only rated as low (3.3), we wanted to solve this issue quickly. We also found a bug where we would terminate the session and delete it from the database table. But when we did a query, we would make sure to apply newer events to the found session (using the event sequence of the found session). At first glance this totally makes sense, because the projection might be a little behind and a new session might not even be in the database, yet. The problem was that exactly this piece of code did not handle the logout event. This was an [easy but important fix](https://github.com/zitadel/zitadel/releases/tag/v1.10.5) we provided a couple of days after the security test. It also demonstrated to us that we need more automated tests on the query side of our APIs.

## What we are planning to do

We have a fast forward strategy and while we’re releasing a lot of new features frequently , we need to keep code and application security in mind. Therefore, we want to challenge our security from the outside more often and plan to do such security audits at least every six months. Moreover we have a [basic responsible disclosure policy](https://github.com/zitadel/zitadel/blob/main/SECURITY.md) that we want to expand further over the near future, most likely coupled with an official bug bounty program.

Furthermore we will have to use the potential we gain through our CQRS / Eventsourcing architecture and generate use case based reports, e.g. present a user not only failed login attempts but also other relevant events that have taken place since the last successful login, such as gaining or losing access to a project and so on.

And last but not least, we are going to push passwordless authentication (WebAuthN) further as this solves a lot of problems regarding compromised or weak passwords.


During the first half of 2021 an external specialist conducted a security analysis of ZITADEL.  We want to show you the results, what we already did and what we’re planning to do in the future.

Pentest Results H1 / 2021


<a href="https://smartive.ch/" target="_blank" rel="noreferrer">
  <img
    src="/blog/2021-10-25-success-story-smartive-bin-01.svg"
    width="100%"
    alt="Logo of smartive AG"
  />
</a>

[smartive](https://smartive.ch/) replaced the existing backend of the bin mobile app for the [bin cooperative’s eLearning platform](https://www.bin.ch/de/leistungen/elearning.html). The app makes teaching materials and books digitally available to apprentices and simplifies teaching thanks to sophisticated tools like highlighting and drawing. They integrated ZITADEL for authentication and authorization to rely on established standards, which saved them at least two weeks of initial development time and even more in maintenance.

## Key outcomes

- Migrated five thousand users to ZITADEL
- Full integration within eight hours
- Predictable pricing independent of user count

## Challenge

smartive had to replace an existing backend, because it was outdated and didn’t rely on standards. The goal was to implement the same functionality as before. Using state of the art technologies and to rely on standards wherever possible was the main focus. Using an external IAM solution was the right choice to minimize development impact. The migration process must be as smooth as possible, because most students already used the first version of the app.

## Solution

In ZITADEL the bin app has one organization that contains projects for the development and productive environment. All the users are managed by the bin organization which means that they have a @bin.ch account for their login. Since these users already existed in the previous system they had to be migrated to ZITADEL.
The books a student is allowed to use are controlled by roles defined on the project and assigned to the user.

Implementing the migration process that imported all users into ZITADEL was an easy task, as the ZITADEL API was straightforward to use. Because ZITADEL follows the API first approach, all the data from the existing system could be migrated with a migration script that was executed once for each environment.

<img
  src="/blog/2021-10-25-success-story-smartive-bin-04.png"
  width="100%"
  alt="Login screen of ZITADEL with bin logo and red colors"
/>

## Result

Right in time for the new semester, smartive was able to deliver the new app. With the integration of an existing IAM instead of a custom implementation they saved several weeks in development time and even more in maintenance. The predictable pricing, independent of user count, and the fast feedback from ZITADEL support contributed to the success of the project.

## Future Plans

smartive is currently working on an extension, such that the bin app becomes a platform for multiple publishers. With multi-tenancy out of the box, ZITADEL is still the perfect match for their future endeavors.

## Testimonials

<img
  src="/blog/2021-10-25-success-story-smartive-bin-02.png"
  width="30%"
  alt="Portrait of Christoph Bühler in a red shirt on blue background."
/>
“I appreciate the proximity to the ZITADEL team, they react quickly and provide competent
support.” Christoph Bühler, Software Engineer

## About the solution

The bin-app is part of the eLearning offering provided by bin-Genossenschaft (Bildungsnetz Schweizer Schreiner). More information and further links to the solution can be found at https://www.bin.ch/de/leistungen/elearning.html

## About smartive

smartive AG  
Pfingstweidstrasse 60  
8005 Zürich  
https://smartive.ch/

- Industry: Software Development, Agency
- Team Size: 22
- Clients: Bin, Migros, Schweizerische Eidgenossenschaft, Frontify
- German-speaking Switzerland

[Download this story.](/blog/2021-10-25-success-story-smartive-bin-03.pdf)


smartive replaced the existing backend of the bin mobile app for the bin cooperative’s eLearning platform with ZITADEL.

How smartive implemented a mobile app with ZITADEL for thousands of students

Our talk at Siemens Open Source event on May 25, 2021

Open Source @ Siemens


## Description

The team behind ZITADEL shares what we developed over the summer, what issues we encountered on the way, and what is planned in the near feature.

<br />
<div className="my-4">
  <YouTubeExtended
    videoId={"b95LK2oiIiw"}
    containerClassName={"youtube-container"}
    stopsTitle="Topics covered"
    stopList={[
      { name: "Team", time: "5:25" },
      { name: "Privatelabeling", time: "8:00" },
      { name: "Passwordless", time: "12:03" },
      { name: "Query Optimization", time: "17:00" },
      { name: "Postmap rings 😂", time: "20:50" },
      { name: "OIDC RP Certification", time: "21:33" },
      { name: "Sentry", time: "26:10" },
      { name: "New Nextjs template / example", time: "32:05" },
      { name: "Quick Demo Privatelabeling in Console", time: "33:12" },
      { name: "Issues and how we solved them", time: "37:05" },
      {
        name: "Upcoming: Extending external identity providers",
        time: "56:05",
      },
      { name: "Next up: What’s coming to ZITADEL", time: "1:01:28" },
    ]}
  ></YouTubeExtended>
</div>

## References

- [Software Architecture / Query Models / CQRS](https://zitadel.com/docs/concepts/architecture/software)
- [Certification OIDC RP Library in Go](https://github.com/zitadel/oidc)
- [Examples / Templates](https://zitadel.com/docs/sdk-examples/introduction)


The team behind ZITADEL shares what we developed over the summer, what issues we encountered on the way, and what is planned in the near feature.

Into the CAOS | No 1.


<div className="my-4">
  <YouTube videoId={"LSrxyu2n51Y"} containerClassName={"youtube-container"} />
</div>

## Description

In this short interview with Gregory Blaser we introduce our consulting partner Blaser Meewes & Partner, in short bmparter. Bmpartner is a boutique-consultancy in Switzerland, with focus on in IT strategy and delivery management. They specialize in large and complex IT projects, IT strategy consulting, and digitalization of sales departments and client facing business units.

Gregory Blaser provides more detail about bmpartner, why identity & access management is relevant for their clients, and how the partnership with CAOS strenghtens the value for clients. He highlights three important points at the end of the inverview for customers that want to tackle their identity & access management challenges: Don't underestimate the challenge and address it early on, break with old patterns whenever possible, and build internal know-how during the process.

## Topics covered

- Introduction (00:00)
- About bmpartner (00:15)
- The ideal client (01:22)
- Relevance of identity & access management (02:17)
- Partnership with CAOS (04:54)
- Important points in IAM projects (06:32)

## bmpartner

[bmpartner](https://www.bmpartner.ch/) is our consulting partner. We work together to deliver complex and large scale IT strategy and implementation projects. CAOS provides additional expertise in the field of identity and access management. [Gregory Blaser](https://www.bmpartner.ch/team/gregory-blaser.html) is managing partner of bmpartner.


We introduce our consulting partner Blaser Meewes & Partner

Introducing bmpartner


IMO = In My Opinion is a blog format where a author reflects his own opinion

At this year's Worldwide Developers Conference (WWDC) 2021 Apple did announce something that really got me thinking about what the future for Risk Based Authentication (RIBA) holds.

In short Apple did announce a feature called “iCloud Private Relay”. What it basically does is to hide the user's origin from the services the user calls. This in a way that Apple and the egress provider should not be able to track the user. Whether that’s true or not is a thing we will see in the future. This design somewhat resembles what TOR does but with less privacy.

<img
  src="/blog/2021-07-07-imo-rba-is-broken-01.png"
  width="100%"
  alt="iCloud Private Relay"
/>

If a user enables this feature, your RIBA seriously will have a bad time. This is because, as you can see below, the user's IP Address will be more or less useless as a signal. As of writing this blog I was in Switzerland and the IP used to egress my traffic was in a region located in the US. If this also tends to change a lot and fast you can basically throw away IP addresses as data of your RIBA.

<img
  src="/blog/2021-07-07-imo-rba-is-broken-02.png"
  width="100%"
  alt="Settings Private Relay"
/>

```json
{
  "ip": "2a09:bac0:205::815:95a",
  "ip_decimal": 55878094235311776708809106431373805914,
  "country": "United States",
  "country_iso": "US",
  "country_eu": false,
  "latitude": 37.751,
  "longitude": -97.822,
  "time_zone": "America/Chicago",
  "asn": "AS13335",
  "asn_org": "CLOUDFLARENET",
  "user_agent": {
    "product": "curl",
    "version": "7.76.0",
    "raw_value": "curl/7.76.0"
  }
}
```

Some of you now might say: “Well we have cookies and browser fingerprinting”.
Yes but with Apple's recent announcement towards improving the privacy features of Safari things are getting harder as we speak. And this does not only apply to Apple but other browsers as well.

So what is the solution to all of this?  
To be frank with you, get rid of your RIBA process and start towards the passwordless journey. It both improves your security and the users' risk quite a lot. If you are interested I dearly recommend that you read my [IMO - Passkey in iCloud Keychain](/blog/imo-passkey-in-icloud-keychain/) as well.

So that’s it? Just throw RIBA away?  
Well there is a nuance to this, which I might encourage you to differ a little from that view. If we are talking about **re-authentication** and **session management**, I still recommend you to use signals you collect about the device and user, for example user-agent version, language, time, TLS version and cipher and so on to prevent things like “session cookie extraction” from the browser.

**But please stop relying on RIBA for the plain authentication of a user!**

## Resources

https://www.apple.com/newsroom/2021/06/apple-advances-its-privacy-leadership-with-ios-15-ipados-15-macos-monterey-and-watchos-8/

https://www.theverge.com/2021/6/10/22526881/apple-icloud-plus-privacy-subscription-services-revenue-wwdc-2021

https://www.heise.de/hintergrund/Die-Kruecke-fuer-Passwoerter-und-wie-sie-ausgenutzt-wird-4970547.html


Apple's “iCloud Private Relay” breaks risk based authentication (RIBA). You should get rid of your RIBA process and start towards the passwordless journey.

IMO - Risk Based Authentication is broken


<div className="my-4">
  <YouTube videoId={"Wly2jqbzP8Q"} containerClassName={"youtube-container"} />
</div>

## Outline of the Talk

- Introduction (00:00)
- Why did we actually start the project (07:24)
  - 2019 - the start (08:00)
  - The basic requirements of most (X)aaS project today (10:04)
  - What generally felt wrong or was missing (13:08)
  - What functions each project needs - sooner or later (28:02)
  - Challenge 1 - Identity Brokering with self-service (28:28)
  - Challenge 2 - Delegation of roles to other organizations (31:24)
  - Challenge 3 - Audit trail (34:15)
  - Challenge 4 - Analytics & Reporting (36:50)
  - Influential products to our vision (38:47)
- What makes ZITADEL so special (39:36)
- What else can ZITADEL do for you (43:40)

## Description

Abstract: The concepts of Authentication and Authorization play an important role for securing services and data. With the emergence of Multi- and Hybrid-Cloud Patterns a new challenge arises for secure Authentication across domains, while privacy of employee or customer information is more important than ever.

In this talk we want to explain some of the key challenges with today’s Identity & Access Management (IAM) and why we ultimately built a new open source IAM, ZITADEL, to change the traditional business-model and principles of IAM solutions for example by

- Optimizing for day-two operations and elasticity with a cloud-native architecture
- Including all security-related functions (like MFA, Passwordless) in the core of the product
- Building a solid audit trail through event-sourcing
- Encouraging automation and easy integration of lifecycle management

You will gain a better understanding of IAM concepts and themes such as

- Federation / IAM Service
- Human to machine interaction (SSO, Passwordless)
- Machine to machine interaction (Service Accounts, Delegation)
- Identity Brokering
- User and access management self-service
- Workflow integration with APIs & Webhooks

Speaker: Florian Forster, CEO of CAOS, is an IT security fanatic with extensive knowledge of electronic identities. Florian participated in the IAM workgroup of eCH, is a member of the OpenID Foundation and headed, prior to starting CAOS, an eGovernment team focussing on IAM topics.

## CH Open & CAOS

CH Open is the Association for the promotion of Open Source Software and Open Standards in Switzerland. CAOS manages the open source products ZITADEL and CAOS, and is member of the association since 2021.

## Further resources

- [CH Open](https://www.ch-open.ch/en/)
- [OSS Directory](https://www.ossdirectory.com/)
- [CH Open Business Lunch](https://www.ch-open.ch/ch-open-business-events/ch-open-business-lunch/)


Our live stream from the CH Open Business Lunch event on June 10, 2021

CH Open Business Lunch - Why we built an open source Identity & Access Management


## Tl;dr

Kubernetes clusters along with all the tooling needed for operating a production grade service should be disposable just like its nodes and pods are disposable. This principle enables you to re-architect your whole infrastructure, solve a lot of your present problems and have a good sleep at night for your services’ full lifecycles.

Treating clusters as cattle enables you to run many small clusters, e.g. one for each domain and environment. As a result, you are comfortable with all day-two operations like keeping the Kubernetes, tooling, operating systems and package versions up-to-date, because you have perfect isolation and similar environments.

In order to achieve this similarity, you choose to follow the GitOps pattern, not only for your service deployments to be reproducible. Also the desired states for the tooling, the Kubernetes installation itself, CPU, memory, networking and storage you just declare in Git and let operator binaries do their work. With comprehensive GitOps, you have your day-one and day-two operations automated. And you get disaster recovery for free.

By relying on Vanilla Kubernetes instead of a complex platform orchestration machinery like OpenShift or Rancher and by having your infrastructure providers abstracted, you are free to choose and instantly change your vendors. If there is something wrong with the pasture grass, you just move your cattle somewhere else.

Having the reproducibility from GitOps combined with the flexibility of abstracted infrastructure providers and Vanilla Kubernetes makes treating clusters as cattle really easy. This way of automation enables you to run many small, isolated, secure and service oriented platforms whenever and wherever you need them. Luckily, we invented this wheel called [ORBOS](https://github.com/caos/orbos) already. It’s Open Source Software and free to use for everyone.

## The Shaping of Our Philosophy

We at CAOS AG founded our startup roughly two years ago. Back then, most of us worked on a self-developed Identity and Access Management System (IAM) at an E-Government company. As this IAM has grown in users and traffic, we found that we could massively improve the quality of our highly critical service, if we operated it ourselves on top of Kubernetes. As our company didn’t have any Kubernetes cluster available at that time, nor any experience with Kubernetes, we created and maintained one ourselves using Tectonic, which back then belonged to CoreOS (now RedHat). There is not that much to bootstrapping a new small cluster and we had a quick performance gain with all the convenience Kubernetes has to offer.

Soon, developers from other domains of the company started deploying their applications on our cluster too. It was inevitable that at some point in time we ran into pod limits and other problems that came with the cluster’s scale. Experiencing these issues from a service provider's perspective made us wonder if there isn’t a better solution. We concluded that dedicating a Kubernetes cluster to a service, including the infrastructure and tooling, improves a provider's ability to guarantee its quality. We saw our chance to go to market with a brand new Cloud Native IAM [ZITADEL](https://zitadel.com/) on our own, distinguishing it from other IAM vendors' products by including the automation for small and dedicated Kubernetes clusters.

If you need arguments for running many small clusters instead of a big one and solutions for doing so without increasing your operational costs, read on.

<div className="team-members">
  <img src="/blog/2021-06-30-cattle-not-pets-03.jpeg" alt="Florian" />
  <img src="/blog/2021-06-30-cattle-not-pets-04.jpeg" alt="Christian" />
  <img src="/blog/2021-06-30-cattle-not-pets-05.jpeg" alt="Silvan" />
  <img src="/blog/2021-06-30-cattle-not-pets-06.jpeg" alt="Fabienne" />
  <img src="/blog/2021-06-30-cattle-not-pets-07.jpeg" alt="Max Peintner" />
  <img src="/blog/2021-06-30-cattle-not-pets-08.jpeg" alt="Stefan" />
  <img src="/blog/2021-06-30-cattle-not-pets-09.jpeg" alt="Elio" />
  <img src="/blog/2021-06-30-cattle-not-pets-10.jpeg" alt="Michael" />
  <img src="/blog/2021-06-30-cattle-not-pets-11.jpeg" alt="Jürg" />
  <img src="/blog/2021-06-30-cattle-not-pets-12.jpeg" alt="Livio" />
  <img src="/blog/2021-06-30-cattle-not-pets-13.jpeg" alt="Max Panne" />
</div>

From left to right: Florian Forster, Christian Jakob, Silvan Reusser, Fabienne Gerschwiler, Max Peintner, Stefan Benz, Elio Bischof, Michael Wäger, Jürg Rinaldi, Livio Amstutz, Maximilian Panne

## Isolate Your Domains

Doing DevOps means that dev and ops engineers have a common goal, which is providing good service quality. The ops side traditionally needs to run many services from many domains with different quality requirements on the same infrastructure and platform. This makes it hard to serve the developers the exact infrastructure they need. Imagine each domain has its own production cluster and a set of test clusters. The clusters are guaranteed to never interfere with each other on the operating system level through NFS volumes, resource consumption and the like.

If the tech stack is siloed along its services all the way down to the infrastructure, not only can each stack be precisely tweaked to the domains and services needs. The tweaker is also more confident in doing so, because he is absolutely sure his tweaks have no unforeseen negative impacts on other domains. If something goes wrong, then not the whole company is at risk that the lights go out, but the blast radius is limited to that domain only. The improved confidence leads to more security updates, feature updates and configuration optimization of the infrastructure, tooling and the API Gateway.

<img
  src="/blog/2021-06-30-cattle-not-pets-01.jpg"
  width="100%"
  alt="Fireball Comparison"
/>{" "}

Fireball_comparison_pic.jpg found on [wikipedia.org](https://en.wikipedia.org/wiki/File:Fireball_comparison_pic.jpg)

## Manage Your Desired States Only

You might be thinking that the operational costs would grow almost linearly with the number of clusters. And that’s true for all tasks that you manually execute. Manual tasks are not only expensive and error prone, they also lead to hardly reproducible states which decreases your confidence and performance when doing day-two operations. So you are forced to automate each of them. Currently, the GitOps pattern is gaining adoption for Kubernetes workload deployment. But it’s very suitable for the platform and infrastructure as well. With GitOps, you focus on automating day-two operations and let a system reconcile its current state based on a declared desired state. If anybody changes something in the system, the reconciling operator reverts the changes immediately and brings the system back to its desired state. So with GitOps you are guaranteed to have a self healing and reproducible system over time.

In [this blog post](https://blog.atomist.com/kubernetes-clusters-pets-or-cattle/), David Dooling from Atomist is advocating the GitOps pattern for solving the reproducibility problem too. But he focuses on storing the Kubernetes resources declarations in Git and relies on managed services and third party CLI tools for bootstrapping new clusters. I think this doesn’t scale well if you want to run many small clusters instead of few big ones. If you need to reproduce a cluster along with its infrastructure, you need the desired infrastructure state to be declared somewhere, preferably in a provider independent way. The other point is, that when storing a mess of hundreds of Kubernetes resources in Git for many clusters in a “copy-and-own-way”, your clusters will start to behave differently from each other over time until their differences are not manageable anymore.

[ORBOS](https://github.com/caos/orbos) fills this gap. Our ORBOS operators interpret their self defined desired states and translate them into cloud and Kubernetes resources. This abstraction layer not only leads to having just a fraction of YAML stored in Git, also the differences in cloud provider infrastructure is as minimal as possible and transparent in their YAML. With ORBOS, you can scale to thousands of clusters without having to employ tons of engineers. [Here](https://github.com/caos/orbos/blob/master/examples/orbiter/gce/orbiter.yml) you find an example YAML for the infrastructure and [here](https://github.com/caos/orbos/blob/master/examples/boom/boom.yml) is a dead-simple one for the platform tooling that omits all default values.

<a href="https://github.com/caos/orbos">
  <img
    className="hidden dark:block"
    src="/blog/2021-06-30-cattle-not-pets-14.png"
    width="30%"
    alt="ORBOS Logo dark"
  />
  <img
    className="block dark:hidden"
    src="/blog/2021-06-30-cattle-not-pets-15.png"
    width="30%"
    alt="ORBOS Logo light"
  />
</a>

## Easily Recover from Disasters

When a cluster goes down, you need to be able to bring back your service’s last state from scratch. To prepare for this, you have to make your workload’s state declarable. This means for example, that all database schema changes should be tracked, tied to your services version and re-executable at any time in an idempotent way. For example, we built a ZITADEL operator that uses [Flyway](https://flywaydb.org/). Check it out on [GitHub](https://github.com/zitadel/zitadel). When your whole workload is declared, you can easily replicate it to another cluster. But in a disaster recovery case where the whole cluster goes down, you might be in trouble getting the declarations out of the broken cluster. Here is where GitOps is helping out again, because with GitOps, you have your desired state declarations stored out-of-band. For learning more about our understanding of using GitOps for workload deployment, checkout [Christian Jakob's blog post](https://zitadel.com/blog/dev-gitops-at-caos/) in German.

Now, combine that with a GitOpsed tooling, platform and infrastructure described in the previous chapter. If done right, a comprehensively “gitopsed” environment is absolutely bulletproof in almost all failure cases. As GitOps comes with perfect reproducibility, you can test if you’ve done it right as much as you want just by replicating your systems. Congratulations, from here on, you can just kill your cattle instead of taking them to a doctor like they were pets. For being bulletproof in not almost all but really all failure cases, read on.

<div className="giphy-wrapper">
  <iframe
    src="https://giphy.com/embed/QMHoU66sBXqqLqYvGO"
    width="100%"
    height="100%"
    frameBorder="0"
    className="giphy-embed"
    allowFullScreen
  ></iframe>
</div>
<p>
  <a href="https://giphy.com/gifs/this-is-fine-QMHoU66sBXqqLqYvGO">via GIPHY</a>
</p>

## Avoiding Lock-Ins

Cattle care less about its owner or territory than pets do. A Cluster on Google Compute Engine should be seamlessly reproducible with other cloud providers and also on-premises if your requirements for prices, data policies, locality etc. change. If you use managed Kubernetes clusters like GKE or AKS, you will have many fancy and integrated features like monitoring or networking. This is comfortable but comes at the cost of a vendor lock-in causing much headache when you try to move to another provider. Especially in cases of critical networking or hardware failures you are glad if you have multiple infrastructure providers abstracted so they are at your disposal immediately.

Also, your workload and Kubernetes API should not rely on the cluster and infrastructure management systems. The operator you use should utilize a generic tool like Kubeadm to maintain the clusters lifecycle and avoid depending on critical API server intermediating components that also need energy and time investments. Without these additional moving targets, you are way more safe from getting locked in to the operator vendor. For example, if you would like to stop using ORBOS and move to another infrastructure and platform management, the only thing you have to do is to delete its deployments. If the operator is stopped or the operator throws errors, neither your service quality nor the Kubernetes API is affected.

<a href="https://kubernetes.io/docs/reference/setup-tools/kubeadm/">
  <img
    src="/blog/2021-06-30-cattle-not-pets-02.png"
    width="20%"
    alt="Kubeadm Logo"
  />
</a>

kubeadm-stacked-color.png found on [kubernetes.io](https://kubernetes.io/docs/reference/setup-tools/kubeadm/)

## Real World Experience

Implementing our GitOps operators definitely took some time and energy. But in return, it enabled us to scale massively and maintain many clusters now. Each cluster is perfectly monitored in a centralized Grafana Cloud instance. If something goes wrong, we fix the operators' code and roll it out to all clusters. Instead of accidentally fixing the same issues again and again, we improved the maturity of our operators incrementally. Now they are in a solid state where they just work and let our customers and us have a good sleep each night.

Treating clusters as cattle enabled us to expand our business model in selling our IAM service [ZITADEL](https://zitadel.com) too. We are not only a Software-as-a-Service provider but also deliver the whole automation needed to run instances dedicated to customers in isolated environments of their choice too. It is crucial for many companies to have full control over all user data stored in our IAM solution, so they often need dedicated instances deployed in their own data centers. An IAM is a mission critical central system that has high quality and scalability requirements to the infrastructure. With our automation, customers yet do not have to assign engineers to the task of operating our product. So the costs that our product causes over time keep being limited and calculable, thanks to our GitOps operators. This gives us a significant advantage over our competitors.

Apart from selling ZITADEL, ORBOS also gave us the opportunity to offer maintenance and support for dedicated Kubernetes clusters. Also, there is a tech company that is running about twenty Kubernetes clusters using ORBOS, split along its domains. Another software agency is running their services on multiple ORBOS managed Kubernetes clusters dedicated to end customers.


I’m sure you know the phrase “treat NODES as cattle, not pets”. It means that we’d rather just replace a “sick” instance by a new healthy one than taking it to a doctor. We take it to another level and say “treat CLUSTERS as cattle, not pets”.

Treat Kubernetes Clusters as Cattle, Not Pets


## Introduction

Last week, something exciting happened: We were actually able to physically attend an event hosted by the [Research Center for Digital Sustainability](https://www.digitale-nachhaltigkeit.unibe.ch/index_eng.html) from the University of Bern. The main reason for the event was the publication of this year’s [Open Source Survey](https://oss-studie.ch/), but for us it was also a great opportunity to learn more about how other companies use or develop Open Source, build communities, and define the cultural aspect around Open Source. And if that was not exciting enough: actually being able to talk to other people over a cold beverage was definitely a big plus after the past months.

I want to share some insights and thoughts about the study, conversations and talks from the event, and finally about our Open\* journey with you.

## Open\*

In the [first talk of the event](https://www.digitale-nachhaltigkeit.unibe.ch/unibe/portal/fak_naturwis/a_dept_math/c_iinfamath/abt_digital/content/e90973/e1045995/e1046107/e1092040/e1092041/01DominicBranderSnowflake-Interoperabilitt_ger.pdf), Dominic Brander introduced us to the term Open\*. That term not only suggests being mindful about including others [1], but also makes it easy to refer to the broader context of Open Source, Open Data, and Open Standards.

I like this term. It makes things easier. And what I realized is that for us, and apparently for others as well, Open Source is not only about letting others see your code, Open Standards is not only about agreeing on a common denominator, and Open Data is not only about getting free data for your school work. It’s much more a cultural thing, for example a culture of trust, enabling collaboration yet not only through interoperability, but of transparency.

I will use the term in this article and most likely in the future.

<sub>
  [1] ‘*’ is also referred to as ‘Gender star’ and is a form to provide gender
  neutrality in a language that otherwise has a grammatical gender.
</sub>

## Insights from Swiss companies

My intention is not to comment on the study, but rather point out some subjectively interesting points for our business and our business area. I realize the study is only available in German, even more motivation to share some insights.

49% of participating companies (N ~150) declare themselves as “heavy users”, using Open Source in more than half of the suggested 28 categories. This figure is up from 29% in 2018. The study offers no more insight on share of total software used within a company, but this is still a good indicator that Open\* solutions are valued broadly.

Interestingly, “Interoperability” is both the top reason FOR and second most important obstacle AGAINST adoption of Open Source in organizations. This point was also discussed by Dominic in his talk: Open* != Interoperability, meaning being an Open* product is not a sufficient condition for being interoperable with every other product. We have to work on interoperability and need to get better at it.

Overall the top 5 objections mentioned against using Open Source:

- Uncertain future of Open Source projects, unclear business model
- Missing integration with other systems
- Lock-ins with current, proprietary system
- Missing features, no matching Open Source alternatives
- Lack of acceptance by end-users, not user-friendly

The authors argue that maintainers/vendors must provide more transparency on the long-term development of their products, coupled with overall more information about the Open Source business model. Likewise the authors urge Open Source communities to become more (end-)customer-centric and generally more accessible.

I fully support that. The same seems to be valid here as for the interoperability argument:  
Releasing your product under an Open License is not sufficient to immediately establish trust and gain a dominant competitive advantage over other solutions out there. Trust and product-leadership also comes from a clear goal with a plan and means to reach and satisfy your users’ needs (aka. “business model”).

Similarly, sharing your project using an Open License and making some processes more transparent out in the open, does not mean these processes are customer-centric. Customer centricity does not mean to collect and work through feedback, but rather to go through a phase of pattern finding, developing innovative ideas for these problem fields, and iterating potential solutions as early as possible with end-users (aka. “innovation process”)

So despite the objections, why are we in the Open\* game?

## Our motivation for Open\*

**Open Standards:**
Thoroughly reviewed, battle-tested protocols and processes are key for developing an application like ZITADEL, that serves as a serious trust anchor and holds sensitive data. We implement open standards wherever possible to guarantee the security and interoperability of our systems.

**Open Source:**
Use of our application is based on trust. Building this trust is demanding, but can be achieved amongst others with transparency and security at the core. Thus our code that runs on our shared-cloud service is shared openly along with our processes, availability strategies and providers (see [Trust](/trust)).

**Open Licence:**
We say “Own your data!”, some others call it “Digital sovereignty”. We both mean the same thing: Reducing vendor lock-in, dependence on a single platform, and the ability to process and store the data wherever you want - even without commercial support. Our software is licensed under Apache 2.0.

**Open Platform:**
We want to focus on what we do best, namely developing and operating an identity and access management system for SaaS-providers. Our users should be able to build their use cases on our platform, and integrate their favorite services with it. That’s why we employ an API-first approach and expose all functions via gRCP, REST and Webhooks.

**Open Community:**
We welcome contributions in every shape and form, and appreciate the time people invest. Eventually the discussions, external challenges, contributions to the platform or our business help us to get better every day and be open for new and innovative ideas.

For us the use of Open\* is much more a mindset that is reflected in our vision, strategy and architecture of our products, than just a process to follow or licence to grant.

What I realized while reflecting on this article is that we might want to figure out how we want to deal with Open Data in the future.

Now to the party-crasher: Sure, we can talk about Open Licenses etc, but should you trust us not pulling a [MongoDB license stunt](https://www.mongodb.com/licensing/server-side-public-license/faq)?

## Thoughts on SSPL license

Another particularly [interesting talk](https://www.digitale-nachhaltigkeit.unibe.ch/unibe/portal/fak_naturwis/a_dept_math/c_iinfamath/abt_digital/content/e90973/e1045995/e1046107/e1092040/e1092047/07SimonSchlauriRonzaniSchlauri-SSPL_ger.pdf) at the event was held by Simon Schlauri on the Server Side Public License (SSPL) and its implications. The situation is as expected: Organizations can publish their code under an SSPL license, so that a SaaS offering would be conditional to a copy-left license. This is concerning for many users who are not the main target of this license (ie. ‘Big Tech’). Lawyers already found clever ways to circumvent the copy-left part, if you still don’t want to pay a license. Frankly, it’s a messy business.

We have discussed the different options for licenses and decided for an Apache 2.0 license. Our business model is inspired by companies like Gitlab that are known for their great Open Source community and through that also being successful as a company.

## Does Open\* have any value, then?

Just right away - the thoughts mentioned above have non-monetary value on their own. Both for us and our users. Trust in our product and our team holds for us as a company and maintainer the greatest value.

We earn our salaries by offering ZITADEL, our identity and access management platform, as a service and charging a monthly subscription for support and some non-security related features. Moreover we offer managed instances or 3rd level support for customer instances. Yet you can use ZITADEL completely for free either by running it yourself or by using our free tier. Keeping up to date with security risks and operating the system in more complex and highly scalable scenarios (e.g. geo-redundant or hybrid setups), that is where we excel through our knowledge and experience. Hence our slogan “always run a changing system”. And ,at least for now, customers are happy to pay for this.

Regarding company value: From my experience until now I can say that neither the tax office nor investors make a big difference between open and closed source. Both parties are whether you have the ability to generate sustainable cash flows with the product. Sure, a patent would help to secure some kind of competitive advantage for a certain time, but in the SaaS world the proof lies in the scalable customer base.

During the event Peter Balsiger also remarked in [his talk](https://www.digitale-nachhaltigkeit.unibe.ch/unibe/portal/fak_naturwis/a_dept_math/c_iinfamath/abt_digital/content/e90973/e1045995/e1046107/e1092040/e1092044/04PeterBalsigerRedHat-CommunityEnterprise_ger.pdf) that IBM paid $34bn in 2018 to acquire Red Hat. That has been by far the biggest deal yet, but 2018 was really a [breakthrough year](https://www.wired.com/story/why-2018-breakout-year-open-source-deals/) regarding Open Source deals, affirming that Open Source companies are valuable.

## Afterthought: Contributing to Open Source as non-techie

After the end of the official event, I had a great conversation with Jannis Valaulta. You can check out [his talk](https://www.digitale-nachhaltigkeit.unibe.ch/unibe/portal/fak_naturwis/a_dept_math/c_iinfamath/abt_digital/content/e90973/e1045995/e1046107/e1092040/e1092042/02JannisValaultaAdfinis-DigitaleSouveranitat_ger.pdf) on digital sovereignty (“Own your data!”, remember?).

I posted the question how “non-techies” could approach contributing to Open Source. For me, as someone with an economics rather than engineering background, the contribution to Open Source projects seemed limited, since “I can’t contribute to fixing or improving stuff”. Other areas as Open Data seem much more approachable for me.

His recommendation was to start with contributing in your area of expertise, optimally in an upcoming hackathon. As an example a designer could contribute with graphics, logos or websites during the hackathon. Similarly, someone with an UX background could help to improve acceptance of users, and someone with a business background could support with figuring out the economic feasibility of the solution for both, customers and providers.

That suggestion seems trivial, but for me that was not so explicitly clear. I realize from my own experience that you most likely can’t excel in all areas that are required to operate a successful Open Source business and outside suggestions are always appreciated. So I think, the next time I might just offer my support, being also open to receive a “no thanks”.

## Great, now what?

If you want to learn more about the study and the talks:

- [View](https://oss-studie.ch) and download the study (German)
- [View](https://www.digitale-nachhaltigkeit.unibe.ch/dienstleistungen/auftragsforschung/oss_studie_2021/index_ger.html) and download all slides and view the talks (German)
- Checkout the Association for the promotion of Open Source Software and Open Standards in Switzerland
- Open Source [activity tracker](https://ossbenchmark.com/) of Swiss companies
- [Directory](https://www.ossdirectory.com/) of Swiss open source solutions

_Unfortunately the event was hosted in German only. Translating the website worked for me partially. Let me know in case you are interested but need help with the language part._

If you’re interested in our projects:

- Checkout our [Github](https://github.com/caos) with the two main projects [ZITADEL](https://github.com/zitadel/zitadel) and [ORBOS](https://github.com/caos/orbos)
- Do you want to share or ask something? [ZITADEL Discussions](https://github.com/zitadel/zitadel/discussions) and [ORBOS Discussions](https://github.com/caos/orbos/discussions)
- Can you [help us get better at engaging](https://github.com/zitadel/zitadel/discussions/1919) our Open Source community?
- Do you have [suggestions on how to make it easier to contribute](https://github.com/zitadel/zitadel/discussions/1920) to our project?
- Want to start contributing? Grab a [“good first issue”](https://github.com/zitadel/zitadel/issues?q=is%3Aissue+is%3Aopen--+label%3A%22good+first+issue%22) and checkout [how to contribute](https://github.com/zitadel/zitadel/blob/main/CONTRIBUTING.md)


What is our motivation for Open Source? Does it bring value to our business? Allow me to share some thoughts on an Open Source study and motivation for Open* with you.

The value of Open*


IMO = In My Opinion is a blog format where a author reflects his own opinion

With the recent announcement of Apple at their annual Worldwide Developers Conference (WWDC) 2021, our perception of how secure authentication should work will gradually shift towards a passwordless future.

The new(est) innovation from Apple to get rid of passwords and second factors is called “Passkeys in iCloud Keychain”. It basically works with the WebAuthN API by utilizing the FIDO2 specification (CTAP2) to create a private / public key pair for each login system. This is normally identified by its domain name of the server challenging your device for authentication. For this to work across devices it is necessary to synchronize the private key in some way.

[Edit: The video has been removed from youtube. You can watch it on developer.apple.com](https://developer.apple.com/videos/play/wwdc2021/10106/)

But wait … synchronizing a private key across your devices with a cloud solution does not sound exactly … sane!  
Well, ok. Let's argue for a minute what the “normal” user does anyway. Many users today rely on password managers to synchronize their passwords and sometimes even their second factors. By doing so, they already expose their secrets to some provider, for example Google, Apple, Microsoft, 1Password, and so on. Even if those providers claim that the secrets are encrypted, it is not a penny better from a security perspective than the synchronization of a private key with the same process. In the end, all private keys, passwords and OTP seed secrets are all **secrets** which allow access to your resources.

But then again, some of you might argue: “I totally control the sync process with a software I run on my own!”  
Well that does not really change the statement above because you end up using a device from one of those vendors anyway and most of them are closed source, so you have absolutely no guarantees that they don’t steal your credentials on the OS level anyway (some exceptions apply - looking at you Linux and to some degree Android).  
To make things worse on an OS or even browser level. Something, like a malicious browser extension or trojan horse, could steal things like access_token or cookies, which broadens the threat model a lot towards OS and browsers.

Oh and to make things worse… Out of band processes like OTP, SMS-TAN and email codes don’t help this effort. If you gain access to either the user's device, OS, or browser, then your whole security model is broken anyway. This is because a lot of today's users just use one device in which they login and also store, receive or generate their second factor.

Ah and yes don't even get me started about those pesky banking authentication apps!

So with this in mind is it still a bad thing to sync the private key?

Well, in my opinion, even with all the downsides like synchronization of private keys (secrets in general) we still gain an **improved user experience** and **phishing resistance**, which I find important these days **without any additional risk** (remember you have them anyway). Also, we reduce the attack surface by **preventing credential reuse** by the user.

So if you ask me: The tradeoff made here by Apple does outweigh the risk. And if you think this risk is too high for you, then just do not use iCloud Passkeys (and password managers in general) and stick to hardware keys.  
If you really care about security you would end up with mTLS (mutual TLS) anyway because this is the only way to mutually authenticate connections and mitigate Man-in-the-middle risk more or less completely (which FIDO2 does not). But this is a topic for another time.

If you want to test iCloud Passkeys today you are best off to use iOS 15, iPadOS 15 or macOS Monterey. Optimally you have multiple devices to see whether the keys get synced across your devices with iCloud Keychain.

As of today more and more systems and services offer the use of passwordless authentication methods. For example your Microsoft Login can be used in this way. Popular Open Source IAM Solutions like Keycloak also give you the option to use passwordless. Our IAM as a Service project ZITADEL does also provide support for the password authentication future.

But remember: Passkeys in iCloud Keychain are **not production ready yet** (technology preview) and are intended for testing only. So for your own sake, please register a fallback hardware key like the ones from our friends at SoloKeys or YubiKey. We recommend you to register multiple keys anyway.


Apple spearheads the push to a passwordless future. But is their proposal to sync your private keys with their iCloud a good idea?

IMO - Passkey in iCloud Keychain


## Introduction

As you have seen from the first part, ZITADEL runs on Kubernetes. You can use any available cluster with the minimal requirements needed to run ZITADEL (SMTP, SMS-Service, Database, Domain), and get started quickly with [ZITADELCTL](https://github.com/zitadel/zitadel/releases/latest). Use [ORBOS](https://github.com/caos/orbos), maybe tailored to use **your** favorite tools, to automate most of the infrastructure and tools.

Where the goal of the [first part](/blog/where-to-start-running-zitadel-1) of this article was to give you some guidance where to start with try or run ZITADEL on your own, in this sequel I want to give you some more insight how we run our ZITADEL SaaS.

## Our current infrastructure & tools

The current infrastructure is centered around locations in Switzerland and for this we use GCE’s compute instances to then manage everything on top through our product [ORBOS](https://github.com/caos/orbos).

As the following components are changeable and this is just our preferred toolset, if anyone needs or wants to use something different, this should be possible without any issues, just ignore our recommendations.

So we have the managed VMs, we use the combination [NGINX](https://www.nginx.com/) and [firewalld](https://github.com/firewalld/firewalld) to handle connections on the node and routing outside the cluster, Kubernetes with [Calico](https://github.com/projectcalico/calico), [Ambassador](https://github.com/datawire/ambassador) inside the cluster with NodePorts to provide connectivity and routing on the inside. Backups are currently made to Google Storage Buckets with CockroachDB native full backups, which makes the restore functionality easier to maintain. To hold metrics and logs to conveniently operate ZITADEL or any other tool we use, there also runs [Loki](https://github.com/grafana/loki) and [Prometheus](https://github.com/prometheus) inside the cluster and the visualization and alerting is possible through [Grafana](https://grafana.com/). We also use some operators that help us to maintain our tools, but maybe this will be a topic for another post.

## Why our Stack

So to just visualise our stack that was described in the section before:

<img
  src="/blog/2021-06-02-where-to-start-running-zitadel-2-01.png"
  width="100%"
  alt="our stack to run ZITADEL"
/>

As base we use VMs of the Google Cloud Engine because we can limit the region and zones so that we can hold our data still in Switzerland but at the same time we can use multiple zones for high availability. The usage of other hyperscalers where you can limit the resources to Switzerland would also be possible, but as we had prior experience and it covered everything we needed, we decided to go with Google Cloud initially.

We decided to make our own Kubernetes automation to provide a platform as a decentralised solution in combination with the advantages of Kubernetes, the usage of already available and tested tools and possibilities of GitOps.

As we want to use already existing tools, one part of ORBOS is the deployment of a set of tools in defined versions with additional integration of the components themselves, for example the scraping of all metrics of the toolset. The result is that there exists one file that contains all necessary configuration for the whole toolset.

Which results in a constellation on the cluster as follows:

<img
  src="/blog/2021-06-02-where-to-start-running-zitadel-2-02.png"
  width="100%"
  alt="our cluster to run ZITADEL"
/>

As our CDN/DNS we use [Cloudflare](https://www.cloudflare.com/) to register subdomains and define firewall-rules, the connections to the subdomains end up on the Ambassador in-cluster and will get routed to the ZITADEL pods. The database we currently use is CockroachDB, as mentioned before, and all backups are made into a Google Cloud Bucket.

For monitoring and alerting we use the combination of Loki (log persistence), Prometheus (metrics persistence) and Grafana (visualisation and alerting) as this combination is widespread and battle-tested, this components could be changed to other solutions but for our goal to be decentralized and low resource consumption we are really comfortable.

## Planned changes in the near future

To further improve the functionality of ORBOS we have different things in the pipeline. One of these is the possibility to use VMs of different zones on Google Cloud to have more options to provide high availability, another is the addition of different implementations of storage options to use for backups. As well as improvements on our own tooling, the current used version for Kubernetes and the used tools will get routinely updated and moved to the newest version. The operators will get new releases as well as the IAM gets new releases, where the operators will as mentioned self-reconcile if you just change the version in the configuration.

## Make or buy?

As you just read and also see in our other blog post “[Always run a changing system - The CAOS approach](/blog/always-run-a-changing-system)” we are all for it to use already implemented tools to make our work easier, but we automate everything we can so we don’t have to do thing multiple times by hand. At the same time we also have to maintain our own code with new releases from the used tooling.

There are advantages and disadvantages to this pattern, but currently the advantages overweigh. As you say “horses for courses”, we use tooling for tasks that are uniformly done for applications, but provide operators for tasks that are specific to ZITADEL, which also integrate the tools with each other.


Part (2/2) As you have seen from the first part, ZITADEL runs on Kubernetes. You can use any available cluster with the minimal requirements needed to run ZITADEL (SMTP, SMS-Service, Database, Domain), and get started quickly with ZITADELCTL. Use ORBOS, maybe tailored to use your favorite tools, to automate most of the infrastructure and tools.

Blog.