
Most of the common Identity and Access Management solutions have a pricing model based on the amount of users.

In our opinion this model leads to shortcuts in security, such as account sharing amongst users, which should be avoided for higher security and traceability. Also, we believe that security features should just be available and should not cost you extra.

That’s why we decided to develop a price model, where you pay per organization, for the degree of customization, and the guaranteed service level agreement (SLA).

## What does that mean?

In our public cloud service we configured some default settings, which we believe make sense for most of the clients. You can create a single or multiple organizations for which you can choose an individual pricing tier. With higher tiers the organization has more configuration options and a higher guaranteed SLA.

- What you get with every tier of the cloud service:
- Unlimited number of users
- Unlimited number of projects and applications
- 24x7 operations
- Access to our APIs

And a variety of important features such as 2FA/MFA, Passwordless, M2M, Identity brokering etc. are included in all tiers. ZITADEL is a bit different here than what you might be used to from other providers. For security features, we make recommendations through default settings. This might lead in some cases to the fact that you may need to upgrade your subscription to get more customization and thus be able to diverge from best practices.

Lastly you benefit from our [security](https://zitadel.com/pricing#Security) measures that we implement to protect our services.

Let’s go through the different tiers that are available as public cloud service.

## Free Tier

The free tier is perfect when you just start developing your application and like to integrate authentication for no budget. You are still in your local environment and you can test all the features and get a feeling on how ZITADEL works. The thing you are missing in this tier is customization options for your login flow and SLA, but at this point of your project this will probably not be relevant.

## Why upgrade to Outpost?

As soon as your project gets more concrete and may already be in a test environment an upgrade makes sense. You will be able to add some customization and we will also help you with our email support. At this point also the SLA of 99.50% is included.

Some of the features you’ll get that may be important at this point:

- Custom Domain verification: Verify your domain so your users can use it for the login name (e.g. road.runner@caos.ch)
- Private Labeling: Customize your login and emails by uploading your logo and font and configure your corporate colors
- Identity Providers: Change the default configuration and configure your own identity providers, so your users can use an existing account (e.g Microsoft, Azure AD, etc.)
- Audit Log retention for one month

## Why upgrade to Starbase?

Your application is almost finished and you are planning to go live. At this point you should think about how much support and SLA is important. We will provide you with a private chat channel to get in touch with us for your support request where you will have a response time of four hours as well as the SLA will be increased to 99.90% availability.

Starbase will also allow you to configure some more customizations:

- Private Labeling: Customize the messages your users get on initialization, password reset, email verification, etc.
- Add some custom metadata values to your users (Key-Value Store)
- Lockout Policy: Change the user lockout settings
- Audit Log retention for seven months

## Why upgrade to Fortress?

This tier is for customers who require reliable and fast support response. We give you the warranty of responding within an hour and a phone number on which you can reach us. Also if your application should have high availability you should think about upgrading because the SLA will go up to 99.95%.

Moreover this tier gives you the maximum of customization and personalization that ZITADEL can offer. For example you can overwrite any of the texts in the login interface and use all available settings for your organization.

This tier will also include features that we feel cater rather towards legacy implementations, which are typically used in enterprise applications. SAML 2.0 Identity brokering, which will be available early next year, is one of those features that is included only in tier.

You also get 13 months of audit log retention, which is important in case you actually are affected by a breach and need to investigate. You will also benefit from advanced reporting and analytics.

In short: You’ll get everything that we can provide - personal support and all features.

## Why would you need a dedicated instance?

We offer a lot in the public cloud instance. You are, however, constrained by the limitations that come with a shared system.

These limitations include on the one hand performance considerations. Your traffic is subject to our rate [limit policy](https://zitadel.com/docs/legal/rate-limit-policy). We always monitor the performance of our systems to guarantee best availability to all customers, but still you might have to deal with ‘noisy neighbours’.

In the future this may be further mitigated by further technical measures to guarantee Quality of Service, especially in the higher-priced tiers, or certain performance packages for individual organizations.

Still, further constraints might include location of your data, infrastructure provider requirements, flexibility in updates and backups, or concerns over performance. That’s where our dedicated instances come into play.

A dedicated instance is a complete ZITADEL system in which you can decide everything yourself. You decide where this system is operated, on which domain it runs, when it should be updated, and which default settings should be stored.

## When do I need more Organizations?

If you have a B2C case you will probably be fine with only one organization. In this organization you will maintain all your projects and applications as well as all your users. In this case you can decide which tier suits best for your project and upgrade on your organization.

But what if you have a B2B case?  
In a B2B case our recommendation is to create an organization for yourself and your project and some for each of your customers. To decide which tier suits your own organization just follow the questions above.

Now each of your customers can decide if they are fine with the default settings (free tier) or if they need some customization. If they need some customization they can decide by themselves which tier suits them. If your customer already has its own organization you can just grant your project to the existing one, that’s managed by your customer.

Having an organization for each of your customers enables you to delegate the authorization management to them. You can find a simple example of a customer portal with delegated authorization on our [GitHub repository](https://github.com/zitadel/zitadel-nextjs-b2b).

The following chart should provide you some visual help on how to structure your organizations, depending on your use case.

<img
  src="/blog/2021-11-24-why-should-I-upgrade-my-zitadel-01.png"
  width="100%"
  alt="Flowchart explaining how you can structure your ZITADEL organizations"
/>


Most of the common Identity and Access Management solutions have a pricing model based on the amount of users. In our opinion this model leads to shortcuts in security.

Why should I upgrade my ZITADEL?


In our ongoing efforts to ease the integration of applications to ZITADEL we already certified the first Go library for OpenID Connect for relying party usage back in July 2021.
And today we are happy to announce that ZITADEL has been officially certified by the OpenID Foundation for the “Basic OP” profile.

These certifications prove that ZITADEL implements the OpenID Connect protocol according to the specifications and that your relying party is guaranteed to work, if it follows the specification.

If you are interested in more details you can checkout ZITADEL [here](https://github.com/zitadel/zitadel), our Go library [here](https://github.com/zitadel/oidc) or visit the [OpenID Certification page](https://openid.net/certification/)


We are happy to announce that ZITADEL has been officially certified by the OpenID Foundation for the “Basic OP” profile.

OpenID Connect Certification


## Description

We were busy since our last Into the CAOS and want to share some updates around ZITADEL and what is planned in the near future.

We will also talk about new features such as Metadata, Actions, new policies as well as what did not go so well.

<br />
<div className="my-4">
  <YouTubeExtended
    videoId={"RZJkNX7s7Zk"}
    containerClassName={"youtube-container"}
    stopsTitle="Topics covered"
    stopList={[
      { name: "Highlights from the past weeks", time: "1:50" },
      { name: "Metadata", time: "5:20" },
      { name: "Handling External Identity Providers", time: "7:06" },
      { name: "Actions", time: "9:00" },
      { name: "Lockout policy", time: "17:00" },
      { name: "Query Side", time: "21:06" },
      { name: "Question to Lockout policy", time: "25:21" },
      { name: "Native Backup", time: "28:19" },
      { name: "OpenAPI/Swagger", time: "30:10" },
      { name: "Issues and learnings", time: "33:29" },
      { name: "New languages", time: "39:38" },
      {
        name: "How to use ZITADEL for Product vs. Organizations",
        time: "42:36",
      },
      { name: "Roadmap", time: "54:31" },
      { name: "Penetration test", time: "58:40" },
    ]}
  ></YouTubeExtended>
</div>

## References

- [Swisscom Cybersecurity Startup Challenge](https://www.swisscom.ch/en/magazine/start-up/startup-challenge-security/)
- [DINAcon Awards](https://awards.dinacon.ch/gewinner2021/)
- [API Documentation](https://zitadel.com/docs/apis/introduction/)
- [Internationalization Documentation](https://zitadel.com/docs/guides/manage/customize/texts#internationalization)
- [Smartive - Built with ZITADEL](https://zitadel.com/blog/success-story-smartive-bin)
- [Pentest results](https://zitadel.com/blog/pentest-results-h1-2021)

## Discuss features

- [Discussion: Metadata on Orgs](https://github.com/zitadel/zitadel/discussions/2454)
- [Discussion: Project settings](https://github.com/zitadel/zitadel/discussions/2263)


We were busy since our last Into the CAOS and want to share some updates around ZITADEL and what is planned in the near future.

Into the CAOS | No 2.


During the first half of 2021 an external specialist conducted a security analysis of ZITADEL. We want to show you the results, what we already did and what we’re planning to do in the future.

## TL;DR

The overall result was rated as good with only one high issue, which we already mitigated partially. Other issues were minor but nonetheless important for us and were fixed within a few days. Others still need more thinking from our side to find a solution that not only solves the immediate risk, but also gives the users a real benefit. Until then a lot of the findings can be mitigated from the users themself by using either Passwordless (WebAuthN) or at least by protecting their accounts with a second factor (2FA).

## Findings

In the following section I will give you a short summary and comment on the highest ranked issues. The number indicates the external assessment according to the Common Vulnerability Scoring System (CVSS).
[You can find the full report here.](/blog/2021-11-11-pentest-results-h1-2021-01.pdf)

### 8.1 Data not Encrypted at Rest (CVSS 8.2 / High)

> “The ZITADEL Console web application does not encrypt data at rest. An attacker with access to the database, can access the stored secret and personal identification information data.”

Although ZITADEL does not encrypt its database, the storage (hard disks) is encrypted. Furthermore, ZITADEL either hashes (bcrypt) or encrypts (AES) certain events for example passwords, OTP Secrets, private keys or temporary codes.

### 8.2 Failed Login Attempts Not Displayed (6.5 / Medium)

> “The ZITADEL Console web application does not display failed login attempts after a successful authentication. An attacker may profit of this missing control, since a user will not know, that he was subject to an attack.”

We certainly lack a corresponding display and we’re planning to have such in the near future. For the moment a user is able to see these and all other events that have occurred on his user on the “Personal Information” page in the “Activity” section:

<img src="/blog/2021-11-11-pentest-results-h1-2021-02.png" width="60%" alt="" />

To mitigate the risk of an attacker attempting to log into your account, we highly recommend using Passwordless (WebAuthN) or at least protect your account with a second factor (2FA) for example Universal Second Factor (U2F) or One Time Password (OTP). ZITADEL allows managers to enforce the use of multi factor or passwordless authentication.

### 8.3, 8.4, 8.6 Missing Protections for Username / Password Authentication

I would like to take a look at the three points together, because the focus is on protecting authentication using usernames and passwords. To say it in advance, these can be mitigated considerably by using WebAuthN or at least a 2FA mechanism:

- 8.3 Missing Protection Against Online Guessing (6.5 / Medium):
  > “The ZITADEL Console web application does not protect user accounts with an application layer mitigation mechanism against automated brute-force attacks. An attacker with access to the login page can perform an online brute-force attack to guess a user’s password.”
- 8.4 Consider Restriction of Common or Compromised Passwords (6.5 / Medium):
  > “The ZITADEL Console web application does not offer the option for IAM Administrators to restrict the usage of common and compromised passwords. An attacker can be successful using a wordlist based brute force attack to takeover user accounts.”
- 8.6 Username Enumeration Possible (4.3 / Medium):
  > “The ZITADEL Console web application login allows enumeration of usernames, due to a warning that is showed after entering a non-existing username. An attacker with access to the login page can perform an online brute-force attack to enumerate existing usernames.”

To mitigate against brute-force attacks at least on a network layer, we use an IP-based rate limiting for our cloud service and dedicated instances. While we also provide a possibility to create a lockout policy, we do not recommend using it, as it also enables attackers to Denial-of-Service a user. Therefore a reasonable solution would be to use Captchas and prompt users to solve it when a user fails to enter a correct password multiple times.

When using a weak password an attacker's chance to take over an account is even higher. Giving organisation administrators the possibility to restrict certain passwords or make users aware that the password they use, might be weak or compromised will be an enhancement we plan to implement in the future.

We’ve discussed a lot whether we should show a warning that the entered username does exist or not. But the fact that a user can authenticate using external identity providers (e.g. Google) somewhat made it obsolete, because when the user enters their username, we might have to redirect them to their identity provider, which will disclose the fact that the user exists. Moreover, a malicious user who already has access to ZITADEL might gain access to this information anyway (e.g. through the client application).

### 8.5 IAM Policies Not Enforced (5.3 / Medium)

> “The ZITADEL Console web application fails to disable user registration and password
> authentication on the server-side. An attacker who knows the technical details, on how to
> register a user, can register a user without access to any projects or organization
> information.“

A registration of a user, which would not have been allowed is certainly a problem and we fixed it. However, there are possible measures and facts that mitigate this issue. First of all a successful registration only allows a user to authenticate, which he might already be able to do so in another organisation, but no authorization to any project is granted by default. So if applications do their supposed authorization checks, they will not have a problem. As the user will not get any manager role within ZITADEL, he will as well not gain access to any information of other users of his organization.

### 8.7 Verbose Error Messages (4.3 / Medium)

> “The ZITADEL Console web application shows verbose error messages in the HTTP Header Grpc-Message. This information can be helpful for an attacker to mount further attack steps.”

Due to the fact that ZITADEL is OpenSource, a lot of this information can be obtained anyway. But we definitely see room for improvement here, especially from a usability aspect and we will continuously improve this.

### 8.8 Consider Usage of Refresh Token (3.3 / Low)

During the security analysis, refresh tokens could not be tested because we were still implementing them. We released this feature the following week (see [v1.8.0](https://github.com/zitadel/zitadel/releases/tag/v1.8.0))

### 8.9 Logout Does Not Terminate Session (3.3 / Low)

Although this was only rated as low (3.3), we wanted to solve this issue quickly. We also found a bug where we would terminate the session and delete it from the database table. But when we did a query, we would make sure to apply newer events to the found session (using the event sequence of the found session). At first glance this totally makes sense, because the projection might be a little behind and a new session might not even be in the database, yet. The problem was that exactly this piece of code did not handle the logout event. This was an [easy but important fix](https://github.com/zitadel/zitadel/releases/tag/v1.10.5) we provided a couple of days after the security test. It also demonstrated to us that we need more automated tests on the query side of our APIs.

## What we are planning to do

We have a fast forward strategy and while we’re releasing a lot of new features frequently , we need to keep code and application security in mind. Therefore, we want to challenge our security from the outside more often and plan to do such security audits at least every six months. Moreover we have a [basic responsible disclosure policy](https://github.com/zitadel/zitadel/blob/main/SECURITY.md) that we want to expand further over the near future, most likely coupled with an official bug bounty program.

Furthermore we will have to use the potential we gain through our CQRS / Eventsourcing architecture and generate use case based reports, e.g. present a user not only failed login attempts but also other relevant events that have taken place since the last successful login, such as gaining or losing access to a project and so on.

And last but not least, we are going to push passwordless authentication (WebAuthN) further as this solves a lot of problems regarding compromised or weak passwords.


During the first half of 2021 an external specialist conducted a security analysis of ZITADEL.  We want to show you the results, what we already did and what we’re planning to do in the future.

Pentest Results H1 / 2021


<a href="https://smartive.ch/" target="_blank" rel="noreferrer">
  <img
    src="/blog/2021-10-25-success-story-smartive-bin-01.svg"
    width="100%"
    alt="Logo of smartive AG"
  />
</a>

[smartive](https://smartive.ch/) replaced the existing backend of the bin mobile app for the [bin cooperative’s eLearning platform](https://www.bin.ch/de/leistungen/elearning.html). The app makes teaching materials and books digitally available to apprentices and simplifies teaching thanks to sophisticated tools like highlighting and drawing. They integrated ZITADEL for authentication and authorization to rely on established standards, which saved them at least two weeks of initial development time and even more in maintenance.

## Key outcomes

- Migrated five thousand users to ZITADEL
- Full integration within eight hours
- Predictable pricing independent of user count

## Challenge

smartive had to replace an existing backend, because it was outdated and didn’t rely on standards. The goal was to implement the same functionality as before. Using state of the art technologies and to rely on standards wherever possible was the main focus. Using an external IAM solution was the right choice to minimize development impact. The migration process must be as smooth as possible, because most students already used the first version of the app.

## Solution

In ZITADEL the bin app has one organization that contains projects for the development and productive environment. All the users are managed by the bin organization which means that they have a @bin.ch account for their login. Since these users already existed in the previous system they had to be migrated to ZITADEL.
The books a student is allowed to use are controlled by roles defined on the project and assigned to the user.

Implementing the migration process that imported all users into ZITADEL was an easy task, as the ZITADEL API was straightforward to use. Because ZITADEL follows the API first approach, all the data from the existing system could be migrated with a migration script that was executed once for each environment.

<img
  src="/blog/2021-10-25-success-story-smartive-bin-04.png"
  width="100%"
  alt="Login screen of ZITADEL with bin logo and red colors"
/>

## Result

Right in time for the new semester, smartive was able to deliver the new app. With the integration of an existing IAM instead of a custom implementation they saved several weeks in development time and even more in maintenance. The predictable pricing, independent of user count, and the fast feedback from ZITADEL support contributed to the success of the project.

## Future Plans

smartive is currently working on an extension, such that the bin app becomes a platform for multiple publishers. With multi-tenancy out of the box, ZITADEL is still the perfect match for their future endeavors.

## Testimonials

<img
  src="/blog/2021-10-25-success-story-smartive-bin-02.png"
  width="30%"
  alt="Portrait of Christoph Bühler in a red shirt on blue background."
/>
“I appreciate the proximity to the ZITADEL team, they react quickly and provide competent
support.” Christoph Bühler, Software Engineer

## About the solution

The bin-app is part of the eLearning offering provided by bin-Genossenschaft (Bildungsnetz Schweizer Schreiner). More information and further links to the solution can be found at https://www.bin.ch/de/leistungen/elearning.html

## About smartive

smartive AG  
Pfingstweidstrasse 60  
8005 Zürich  
https://smartive.ch/

- Industry: Software Development, Agency
- Team Size: 22
- Clients: Bin, Migros, Schweizerische Eidgenossenschaft, Frontify
- German-speaking Switzerland

[Download this story.](/blog/2021-10-25-success-story-smartive-bin-03.pdf)


smartive replaced the existing backend of the bin mobile app for the bin cooperative’s eLearning platform with ZITADEL.

How smartive implemented a mobile app with ZITADEL for thousands of students

Our talk at Siemens Open Source event on May 25, 2021

Open Source @ Siemens


## Description

The team behind ZITADEL shares what we developed over the summer, what issues we encountered on the way, and what is planned in the near feature.

<br />
<div className="my-4">
  <YouTubeExtended
    videoId={"b95LK2oiIiw"}
    containerClassName={"youtube-container"}
    stopsTitle="Topics covered"
    stopList={[
      { name: "Team", time: "5:25" },
      { name: "Privatelabeling", time: "8:00" },
      { name: "Passwordless", time: "12:03" },
      { name: "Query Optimization", time: "17:00" },
      { name: "Postmap rings 😂", time: "20:50" },
      { name: "OIDC RP Certification", time: "21:33" },
      { name: "Sentry", time: "26:10" },
      { name: "New Nextjs template / example", time: "32:05" },
      { name: "Quick Demo Privatelabeling in Console", time: "33:12" },
      { name: "Issues and how we solved them", time: "37:05" },
      {
        name: "Upcoming: Extending external identity providers",
        time: "56:05",
      },
      { name: "Next up: What’s coming to ZITADEL", time: "1:01:28" },
    ]}
  ></YouTubeExtended>
</div>

## References

- [Software Architecture / Query Models / CQRS](https://zitadel.com/docs/concepts/architecture/software)
- [Certification OIDC RP Library in Go](https://github.com/zitadel/oidc)
- [Examples / Templates](https://zitadel.com/docs/sdk-examples/introduction)


The team behind ZITADEL shares what we developed over the summer, what issues we encountered on the way, and what is planned in the near feature.

Into the CAOS | No 1.


<div className="my-4">
  <YouTube videoId={"LSrxyu2n51Y"} containerClassName={"youtube-container"} />
</div>

## Description

In this short interview with Gregory Blaser we introduce our consulting partner Blaser Meewes & Partner, in short bmparter. Bmpartner is a boutique-consultancy in Switzerland, with focus on in IT strategy and delivery management. They specialize in large and complex IT projects, IT strategy consulting, and digitalization of sales departments and client facing business units.

Gregory Blaser provides more detail about bmpartner, why identity & access management is relevant for their clients, and how the partnership with CAOS strenghtens the value for clients. He highlights three important points at the end of the inverview for customers that want to tackle their identity & access management challenges: Don't underestimate the challenge and address it early on, break with old patterns whenever possible, and build internal know-how during the process.

## Topics covered

- Introduction (00:00)
- About bmpartner (00:15)
- The ideal client (01:22)
- Relevance of identity & access management (02:17)
- Partnership with CAOS (04:54)
- Important points in IAM projects (06:32)

## bmpartner

[bmpartner](https://www.bmpartner.ch/) is our consulting partner. We work together to deliver complex and large scale IT strategy and implementation projects. CAOS provides additional expertise in the field of identity and access management. [Gregory Blaser](https://www.bmpartner.ch/team/gregory-blaser.html) is managing partner of bmpartner.


We introduce our consulting partner Blaser Meewes & Partner

Introducing bmpartner


IMO = In My Opinion is a blog format where a author reflects his own opinion

At this year's Worldwide Developers Conference (WWDC) 2021 Apple did announce something that really got me thinking about what the future for Risk Based Authentication (RIBA) holds.

In short Apple did announce a feature called “iCloud Private Relay”. What it basically does is to hide the user's origin from the services the user calls. This in a way that Apple and the egress provider should not be able to track the user. Whether that’s true or not is a thing we will see in the future. This design somewhat resembles what TOR does but with less privacy.

<img
  src="/blog/2021-07-07-imo-rba-is-broken-01.png"
  width="100%"
  alt="iCloud Private Relay"
/>

If a user enables this feature, your RIBA seriously will have a bad time. This is because, as you can see below, the user's IP Address will be more or less useless as a signal. As of writing this blog I was in Switzerland and the IP used to egress my traffic was in a region located in the US. If this also tends to change a lot and fast you can basically throw away IP addresses as data of your RIBA.

<img
  src="/blog/2021-07-07-imo-rba-is-broken-02.png"
  width="100%"
  alt="Settings Private Relay"
/>

```json
{
  "ip": "2a09:bac0:205::815:95a",
  "ip_decimal": 55878094235311776708809106431373805914,
  "country": "United States",
  "country_iso": "US",
  "country_eu": false,
  "latitude": 37.751,
  "longitude": -97.822,
  "time_zone": "America/Chicago",
  "asn": "AS13335",
  "asn_org": "CLOUDFLARENET",
  "user_agent": {
    "product": "curl",
    "version": "7.76.0",
    "raw_value": "curl/7.76.0"
  }
}
```

Some of you now might say: “Well we have cookies and browser fingerprinting”.
Yes but with Apple's recent announcement towards improving the privacy features of Safari things are getting harder as we speak. And this does not only apply to Apple but other browsers as well.

So what is the solution to all of this?  
To be frank with you, get rid of your RIBA process and start towards the passwordless journey. It both improves your security and the users' risk quite a lot. If you are interested I dearly recommend that you read my [IMO - Passkey in iCloud Keychain](/blog/imo-passkey-in-icloud-keychain/) as well.

So that’s it? Just throw RIBA away?  
Well there is a nuance to this, which I might encourage you to differ a little from that view. If we are talking about **re-authentication** and **session management**, I still recommend you to use signals you collect about the device and user, for example user-agent version, language, time, TLS version and cipher and so on to prevent things like “session cookie extraction” from the browser.

**But please stop relying on RIBA for the plain authentication of a user!**

## Resources

https://www.apple.com/newsroom/2021/06/apple-advances-its-privacy-leadership-with-ios-15-ipados-15-macos-monterey-and-watchos-8/

https://www.theverge.com/2021/6/10/22526881/apple-icloud-plus-privacy-subscription-services-revenue-wwdc-2021

https://www.heise.de/hintergrund/Die-Kruecke-fuer-Passwoerter-und-wie-sie-ausgenutzt-wird-4970547.html


Apple's “iCloud Private Relay” breaks risk based authentication (RIBA). You should get rid of your RIBA process and start towards the passwordless journey.

IMO - Risk Based Authentication is broken


<div className="my-4">
  <YouTube videoId={"Wly2jqbzP8Q"} containerClassName={"youtube-container"} />
</div>

## Outline of the Talk

- Introduction (00:00)
- Why did we actually start the project (07:24)
  - 2019 - the start (08:00)
  - The basic requirements of most (X)aaS project today (10:04)
  - What generally felt wrong or was missing (13:08)
  - What functions each project needs - sooner or later (28:02)
  - Challenge 1 - Identity Brokering with self-service (28:28)
  - Challenge 2 - Delegation of roles to other organizations (31:24)
  - Challenge 3 - Audit trail (34:15)
  - Challenge 4 - Analytics & Reporting (36:50)
  - Influential products to our vision (38:47)
- What makes ZITADEL so special (39:36)
- What else can ZITADEL do for you (43:40)

## Description

Abstract: The concepts of Authentication and Authorization play an important role for securing services and data. With the emergence of Multi- and Hybrid-Cloud Patterns a new challenge arises for secure Authentication across domains, while privacy of employee or customer information is more important than ever.

In this talk we want to explain some of the key challenges with today’s Identity & Access Management (IAM) and why we ultimately built a new open source IAM, ZITADEL, to change the traditional business-model and principles of IAM solutions for example by

- Optimizing for day-two operations and elasticity with a cloud-native architecture
- Including all security-related functions (like MFA, Passwordless) in the core of the product
- Building a solid audit trail through event-sourcing
- Encouraging automation and easy integration of lifecycle management

You will gain a better understanding of IAM concepts and themes such as

- Federation / IAM Service
- Human to machine interaction (SSO, Passwordless)
- Machine to machine interaction (Service Accounts, Delegation)
- Identity Brokering
- User and access management self-service
- Workflow integration with APIs & Webhooks

Speaker: Florian Forster, CEO of CAOS, is an IT security fanatic with extensive knowledge of electronic identities. Florian participated in the IAM workgroup of eCH, is a member of the OpenID Foundation and headed, prior to starting CAOS, an eGovernment team focussing on IAM topics.

## CH Open & CAOS

CH Open is the Association for the promotion of Open Source Software and Open Standards in Switzerland. CAOS manages the open source products ZITADEL and CAOS, and is member of the association since 2021.

## Further resources

- [CH Open](https://www.ch-open.ch/en/)
- [OSS Directory](https://www.ossdirectory.com/)
- [CH Open Business Lunch](https://www.ch-open.ch/ch-open-business-events/ch-open-business-lunch/)


Our live stream from the CH Open Business Lunch event on June 10, 2021

CH Open Business Lunch - Why we built an open source Identity & Access Management


## Tl;dr

Kubernetes clusters along with all the tooling needed for operating a production grade service should be disposable just like its nodes and pods are disposable. This principle enables you to re-architect your whole infrastructure, solve a lot of your present problems and have a good sleep at night for your services’ full lifecycles.

Treating clusters as cattle enables you to run many small clusters, e.g. one for each domain and environment. As a result, you are comfortable with all day-two operations like keeping the Kubernetes, tooling, operating systems and package versions up-to-date, because you have perfect isolation and similar environments.

In order to achieve this similarity, you choose to follow the GitOps pattern, not only for your service deployments to be reproducible. Also the desired states for the tooling, the Kubernetes installation itself, CPU, memory, networking and storage you just declare in Git and let operator binaries do their work. With comprehensive GitOps, you have your day-one and day-two operations automated. And you get disaster recovery for free.

By relying on Vanilla Kubernetes instead of a complex platform orchestration machinery like OpenShift or Rancher and by having your infrastructure providers abstracted, you are free to choose and instantly change your vendors. If there is something wrong with the pasture grass, you just move your cattle somewhere else.

Having the reproducibility from GitOps combined with the flexibility of abstracted infrastructure providers and Vanilla Kubernetes makes treating clusters as cattle really easy. This way of automation enables you to run many small, isolated, secure and service oriented platforms whenever and wherever you need them. Luckily, we invented this wheel called [ORBOS](https://github.com/caos/orbos) already. It’s Open Source Software and free to use for everyone.

## The Shaping of Our Philosophy

We at CAOS AG founded our startup roughly two years ago. Back then, most of us worked on a self-developed Identity and Access Management System (IAM) at an E-Government company. As this IAM has grown in users and traffic, we found that we could massively improve the quality of our highly critical service, if we operated it ourselves on top of Kubernetes. As our company didn’t have any Kubernetes cluster available at that time, nor any experience with Kubernetes, we created and maintained one ourselves using Tectonic, which back then belonged to CoreOS (now RedHat). There is not that much to bootstrapping a new small cluster and we had a quick performance gain with all the convenience Kubernetes has to offer.

Soon, developers from other domains of the company started deploying their applications on our cluster too. It was inevitable that at some point in time we ran into pod limits and other problems that came with the cluster’s scale. Experiencing these issues from a service provider's perspective made us wonder if there isn’t a better solution. We concluded that dedicating a Kubernetes cluster to a service, including the infrastructure and tooling, improves a provider's ability to guarantee its quality. We saw our chance to go to market with a brand new Cloud Native IAM [ZITADEL](https://zitadel.com/) on our own, distinguishing it from other IAM vendors' products by including the automation for small and dedicated Kubernetes clusters.

If you need arguments for running many small clusters instead of a big one and solutions for doing so without increasing your operational costs, read on.

<div className="team-members">
  <img src="/blog/2021-06-30-cattle-not-pets-03.jpeg" alt="Florian" />
  <img src="/blog/2021-06-30-cattle-not-pets-04.jpeg" alt="Christian" />
  <img src="/blog/2021-06-30-cattle-not-pets-05.jpeg" alt="Silvan" />
  <img src="/blog/2021-06-30-cattle-not-pets-06.jpeg" alt="Fabienne" />
  <img src="/blog/2021-06-30-cattle-not-pets-07.jpeg" alt="Max Peintner" />
  <img src="/blog/2021-06-30-cattle-not-pets-08.jpeg" alt="Stefan" />
  <img src="/blog/2021-06-30-cattle-not-pets-09.jpeg" alt="Elio" />
  <img src="/blog/2021-06-30-cattle-not-pets-10.jpeg" alt="Michael" />
  <img src="/blog/2021-06-30-cattle-not-pets-11.jpeg" alt="Jürg" />
  <img src="/blog/2021-06-30-cattle-not-pets-12.jpeg" alt="Livio" />
  <img src="/blog/2021-06-30-cattle-not-pets-13.jpeg" alt="Max Panne" />
</div>

From left to right: Florian Forster, Christian Jakob, Silvan Reusser, Fabienne Gerschwiler, Max Peintner, Stefan Benz, Elio Bischof, Michael Wäger, Jürg Rinaldi, Livio Amstutz, Maximilian Panne

## Isolate Your Domains

Doing DevOps means that dev and ops engineers have a common goal, which is providing good service quality. The ops side traditionally needs to run many services from many domains with different quality requirements on the same infrastructure and platform. This makes it hard to serve the developers the exact infrastructure they need. Imagine each domain has its own production cluster and a set of test clusters. The clusters are guaranteed to never interfere with each other on the operating system level through NFS volumes, resource consumption and the like.

If the tech stack is siloed along its services all the way down to the infrastructure, not only can each stack be precisely tweaked to the domains and services needs. The tweaker is also more confident in doing so, because he is absolutely sure his tweaks have no unforeseen negative impacts on other domains. If something goes wrong, then not the whole company is at risk that the lights go out, but the blast radius is limited to that domain only. The improved confidence leads to more security updates, feature updates and configuration optimization of the infrastructure, tooling and the API Gateway.

<img
  src="/blog/2021-06-30-cattle-not-pets-01.jpg"
  width="100%"
  alt="Fireball Comparison"
/>{" "}

Fireball_comparison_pic.jpg found on [wikipedia.org](https://en.wikipedia.org/wiki/File:Fireball_comparison_pic.jpg)

## Manage Your Desired States Only

You might be thinking that the operational costs would grow almost linearly with the number of clusters. And that’s true for all tasks that you manually execute. Manual tasks are not only expensive and error prone, they also lead to hardly reproducible states which decreases your confidence and performance when doing day-two operations. So you are forced to automate each of them. Currently, the GitOps pattern is gaining adoption for Kubernetes workload deployment. But it’s very suitable for the platform and infrastructure as well. With GitOps, you focus on automating day-two operations and let a system reconcile its current state based on a declared desired state. If anybody changes something in the system, the reconciling operator reverts the changes immediately and brings the system back to its desired state. So with GitOps you are guaranteed to have a self healing and reproducible system over time.

In [this blog post](https://blog.atomist.com/kubernetes-clusters-pets-or-cattle/), David Dooling from Atomist is advocating the GitOps pattern for solving the reproducibility problem too. But he focuses on storing the Kubernetes resources declarations in Git and relies on managed services and third party CLI tools for bootstrapping new clusters. I think this doesn’t scale well if you want to run many small clusters instead of few big ones. If you need to reproduce a cluster along with its infrastructure, you need the desired infrastructure state to be declared somewhere, preferably in a provider independent way. The other point is, that when storing a mess of hundreds of Kubernetes resources in Git for many clusters in a “copy-and-own-way”, your clusters will start to behave differently from each other over time until their differences are not manageable anymore.

[ORBOS](https://github.com/caos/orbos) fills this gap. Our ORBOS operators interpret their self defined desired states and translate them into cloud and Kubernetes resources. This abstraction layer not only leads to having just a fraction of YAML stored in Git, also the differences in cloud provider infrastructure is as minimal as possible and transparent in their YAML. With ORBOS, you can scale to thousands of clusters without having to employ tons of engineers. [Here](https://github.com/caos/orbos/blob/master/examples/orbiter/gce/orbiter.yml) you find an example YAML for the infrastructure and [here](https://github.com/caos/orbos/blob/master/examples/boom/boom.yml) is a dead-simple one for the platform tooling that omits all default values.

<a href="https://github.com/caos/orbos">
  <img
    className="hidden dark:block"
    src="/blog/2021-06-30-cattle-not-pets-14.png"
    width="30%"
    alt="ORBOS Logo dark"
  />
  <img
    className="block dark:hidden"
    src="/blog/2021-06-30-cattle-not-pets-15.png"
    width="30%"
    alt="ORBOS Logo light"
  />
</a>

## Easily Recover from Disasters

When a cluster goes down, you need to be able to bring back your service’s last state from scratch. To prepare for this, you have to make your workload’s state declarable. This means for example, that all database schema changes should be tracked, tied to your services version and re-executable at any time in an idempotent way. For example, we built a ZITADEL operator that uses [Flyway](https://flywaydb.org/). Check it out on [GitHub](https://github.com/zitadel/zitadel). When your whole workload is declared, you can easily replicate it to another cluster. But in a disaster recovery case where the whole cluster goes down, you might be in trouble getting the declarations out of the broken cluster. Here is where GitOps is helping out again, because with GitOps, you have your desired state declarations stored out-of-band. For learning more about our understanding of using GitOps for workload deployment, checkout [Christian Jakob's blog post](https://zitadel.com/blog/dev-gitops-at-caos/) in German.

Now, combine that with a GitOpsed tooling, platform and infrastructure described in the previous chapter. If done right, a comprehensively “gitopsed” environment is absolutely bulletproof in almost all failure cases. As GitOps comes with perfect reproducibility, you can test if you’ve done it right as much as you want just by replicating your systems. Congratulations, from here on, you can just kill your cattle instead of taking them to a doctor like they were pets. For being bulletproof in not almost all but really all failure cases, read on.

<div className="giphy-wrapper">
  <iframe
    src="https://giphy.com/embed/QMHoU66sBXqqLqYvGO"
    width="100%"
    height="100%"
    frameBorder="0"
    className="giphy-embed"
    allowFullScreen
  ></iframe>
</div>
<p>
  <a href="https://giphy.com/gifs/this-is-fine-QMHoU66sBXqqLqYvGO">via GIPHY</a>
</p>

## Avoiding Lock-Ins

Cattle care less about its owner or territory than pets do. A Cluster on Google Compute Engine should be seamlessly reproducible with other cloud providers and also on-premises if your requirements for prices, data policies, locality etc. change. If you use managed Kubernetes clusters like GKE or AKS, you will have many fancy and integrated features like monitoring or networking. This is comfortable but comes at the cost of a vendor lock-in causing much headache when you try to move to another provider. Especially in cases of critical networking or hardware failures you are glad if you have multiple infrastructure providers abstracted so they are at your disposal immediately.

Also, your workload and Kubernetes API should not rely on the cluster and infrastructure management systems. The operator you use should utilize a generic tool like Kubeadm to maintain the clusters lifecycle and avoid depending on critical API server intermediating components that also need energy and time investments. Without these additional moving targets, you are way more safe from getting locked in to the operator vendor. For example, if you would like to stop using ORBOS and move to another infrastructure and platform management, the only thing you have to do is to delete its deployments. If the operator is stopped or the operator throws errors, neither your service quality nor the Kubernetes API is affected.

<a href="https://kubernetes.io/docs/reference/setup-tools/kubeadm/">
  <img
    src="/blog/2021-06-30-cattle-not-pets-02.png"
    width="20%"
    alt="Kubeadm Logo"
  />
</a>

kubeadm-stacked-color.png found on [kubernetes.io](https://kubernetes.io/docs/reference/setup-tools/kubeadm/)

## Real World Experience

Implementing our GitOps operators definitely took some time and energy. But in return, it enabled us to scale massively and maintain many clusters now. Each cluster is perfectly monitored in a centralized Grafana Cloud instance. If something goes wrong, we fix the operators' code and roll it out to all clusters. Instead of accidentally fixing the same issues again and again, we improved the maturity of our operators incrementally. Now they are in a solid state where they just work and let our customers and us have a good sleep each night.

Treating clusters as cattle enabled us to expand our business model in selling our IAM service [ZITADEL](https://zitadel.com) too. We are not only a Software-as-a-Service provider but also deliver the whole automation needed to run instances dedicated to customers in isolated environments of their choice too. It is crucial for many companies to have full control over all user data stored in our IAM solution, so they often need dedicated instances deployed in their own data centers. An IAM is a mission critical central system that has high quality and scalability requirements to the infrastructure. With our automation, customers yet do not have to assign engineers to the task of operating our product. So the costs that our product causes over time keep being limited and calculable, thanks to our GitOps operators. This gives us a significant advantage over our competitors.

Apart from selling ZITADEL, ORBOS also gave us the opportunity to offer maintenance and support for dedicated Kubernetes clusters. Also, there is a tech company that is running about twenty Kubernetes clusters using ORBOS, split along its domains. Another software agency is running their services on multiple ORBOS managed Kubernetes clusters dedicated to end customers.


I’m sure you know the phrase “treat NODES as cattle, not pets”. It means that we’d rather just replace a “sick” instance by a new healthy one than taking it to a doctor. We take it to another level and say “treat CLUSTERS as cattle, not pets”.

Treat Kubernetes Clusters as Cattle, Not Pets


## Introduction

Last week, something exciting happened: We were actually able to physically attend an event hosted by the [Research Center for Digital Sustainability](https://www.digitale-nachhaltigkeit.unibe.ch/index_eng.html) from the University of Bern. The main reason for the event was the publication of this year’s [Open Source Survey](https://oss-studie.ch/), but for us it was also a great opportunity to learn more about how other companies use or develop Open Source, build communities, and define the cultural aspect around Open Source. And if that was not exciting enough: actually being able to talk to other people over a cold beverage was definitely a big plus after the past months.

I want to share some insights and thoughts about the study, conversations and talks from the event, and finally about our Open\* journey with you.

## Open\*

In the [first talk of the event](https://www.digitale-nachhaltigkeit.unibe.ch/unibe/portal/fak_naturwis/a_dept_math/c_iinfamath/abt_digital/content/e90973/e1045995/e1046107/e1092040/e1092041/01DominicBranderSnowflake-Interoperabilitt_ger.pdf), Dominic Brander introduced us to the term Open\*. That term not only suggests being mindful about including others [1], but also makes it easy to refer to the broader context of Open Source, Open Data, and Open Standards.

I like this term. It makes things easier. And what I realized is that for us, and apparently for others as well, Open Source is not only about letting others see your code, Open Standards is not only about agreeing on a common denominator, and Open Data is not only about getting free data for your school work. It’s much more a cultural thing, for example a culture of trust, enabling collaboration yet not only through interoperability, but of transparency.

I will use the term in this article and most likely in the future.

<sub>
  [1] ‘*’ is also referred to as ‘Gender star’ and is a form to provide gender
  neutrality in a language that otherwise has a grammatical gender.
</sub>

## Insights from Swiss companies

My intention is not to comment on the study, but rather point out some subjectively interesting points for our business and our business area. I realize the study is only available in German, even more motivation to share some insights.

49% of participating companies (N ~150) declare themselves as “heavy users”, using Open Source in more than half of the suggested 28 categories. This figure is up from 29% in 2018. The study offers no more insight on share of total software used within a company, but this is still a good indicator that Open\* solutions are valued broadly.

Interestingly, “Interoperability” is both the top reason FOR and second most important obstacle AGAINST adoption of Open Source in organizations. This point was also discussed by Dominic in his talk: Open* != Interoperability, meaning being an Open* product is not a sufficient condition for being interoperable with every other product. We have to work on interoperability and need to get better at it.

Overall the top 5 objections mentioned against using Open Source:

- Uncertain future of Open Source projects, unclear business model
- Missing integration with other systems
- Lock-ins with current, proprietary system
- Missing features, no matching Open Source alternatives
- Lack of acceptance by end-users, not user-friendly

The authors argue that maintainers/vendors must provide more transparency on the long-term development of their products, coupled with overall more information about the Open Source business model. Likewise the authors urge Open Source communities to become more (end-)customer-centric and generally more accessible.

I fully support that. The same seems to be valid here as for the interoperability argument:  
Releasing your product under an Open License is not sufficient to immediately establish trust and gain a dominant competitive advantage over other solutions out there. Trust and product-leadership also comes from a clear goal with a plan and means to reach and satisfy your users’ needs (aka. “business model”).

Similarly, sharing your project using an Open License and making some processes more transparent out in the open, does not mean these processes are customer-centric. Customer centricity does not mean to collect and work through feedback, but rather to go through a phase of pattern finding, developing innovative ideas for these problem fields, and iterating potential solutions as early as possible with end-users (aka. “innovation process”)

So despite the objections, why are we in the Open\* game?

## Our motivation for Open\*

**Open Standards:**
Thoroughly reviewed, battle-tested protocols and processes are key for developing an application like ZITADEL, that serves as a serious trust anchor and holds sensitive data. We implement open standards wherever possible to guarantee the security and interoperability of our systems.

**Open Source:**
Use of our application is based on trust. Building this trust is demanding, but can be achieved amongst others with transparency and security at the core. Thus our code that runs on our shared-cloud service is shared openly along with our processes, availability strategies and providers (see [Trust](/trust)).

**Open Licence:**
We say “Own your data!”, some others call it “Digital sovereignty”. We both mean the same thing: Reducing vendor lock-in, dependence on a single platform, and the ability to process and store the data wherever you want - even without commercial support. Our software is licensed under Apache 2.0.

**Open Platform:**
We want to focus on what we do best, namely developing and operating an identity and access management system for SaaS-providers. Our users should be able to build their use cases on our platform, and integrate their favorite services with it. That’s why we employ an API-first approach and expose all functions via gRCP, REST and Webhooks.

**Open Community:**
We welcome contributions in every shape and form, and appreciate the time people invest. Eventually the discussions, external challenges, contributions to the platform or our business help us to get better every day and be open for new and innovative ideas.

For us the use of Open\* is much more a mindset that is reflected in our vision, strategy and architecture of our products, than just a process to follow or licence to grant.

What I realized while reflecting on this article is that we might want to figure out how we want to deal with Open Data in the future.

Now to the party-crasher: Sure, we can talk about Open Licenses etc, but should you trust us not pulling a [MongoDB license stunt](https://www.mongodb.com/licensing/server-side-public-license/faq)?

## Thoughts on SSPL license

Another particularly [interesting talk](https://www.digitale-nachhaltigkeit.unibe.ch/unibe/portal/fak_naturwis/a_dept_math/c_iinfamath/abt_digital/content/e90973/e1045995/e1046107/e1092040/e1092047/07SimonSchlauriRonzaniSchlauri-SSPL_ger.pdf) at the event was held by Simon Schlauri on the Server Side Public License (SSPL) and its implications. The situation is as expected: Organizations can publish their code under an SSPL license, so that a SaaS offering would be conditional to a copy-left license. This is concerning for many users who are not the main target of this license (ie. ‘Big Tech’). Lawyers already found clever ways to circumvent the copy-left part, if you still don’t want to pay a license. Frankly, it’s a messy business.

We have discussed the different options for licenses and decided for an Apache 2.0 license. Our business model is inspired by companies like Gitlab that are known for their great Open Source community and through that also being successful as a company.

## Does Open\* have any value, then?

Just right away - the thoughts mentioned above have non-monetary value on their own. Both for us and our users. Trust in our product and our team holds for us as a company and maintainer the greatest value.

We earn our salaries by offering ZITADEL, our identity and access management platform, as a service and charging a monthly subscription for support and some non-security related features. Moreover we offer managed instances or 3rd level support for customer instances. Yet you can use ZITADEL completely for free either by running it yourself or by using our free tier. Keeping up to date with security risks and operating the system in more complex and highly scalable scenarios (e.g. geo-redundant or hybrid setups), that is where we excel through our knowledge and experience. Hence our slogan “always run a changing system”. And ,at least for now, customers are happy to pay for this.

Regarding company value: From my experience until now I can say that neither the tax office nor investors make a big difference between open and closed source. Both parties are whether you have the ability to generate sustainable cash flows with the product. Sure, a patent would help to secure some kind of competitive advantage for a certain time, but in the SaaS world the proof lies in the scalable customer base.

During the event Peter Balsiger also remarked in [his talk](https://www.digitale-nachhaltigkeit.unibe.ch/unibe/portal/fak_naturwis/a_dept_math/c_iinfamath/abt_digital/content/e90973/e1045995/e1046107/e1092040/e1092044/04PeterBalsigerRedHat-CommunityEnterprise_ger.pdf) that IBM paid $34bn in 2018 to acquire Red Hat. That has been by far the biggest deal yet, but 2018 was really a [breakthrough year](https://www.wired.com/story/why-2018-breakout-year-open-source-deals/) regarding Open Source deals, affirming that Open Source companies are valuable.

## Afterthought: Contributing to Open Source as non-techie

After the end of the official event, I had a great conversation with Jannis Valaulta. You can check out [his talk](https://www.digitale-nachhaltigkeit.unibe.ch/unibe/portal/fak_naturwis/a_dept_math/c_iinfamath/abt_digital/content/e90973/e1045995/e1046107/e1092040/e1092042/02JannisValaultaAdfinis-DigitaleSouveranitat_ger.pdf) on digital sovereignty (“Own your data!”, remember?).

I posted the question how “non-techies” could approach contributing to Open Source. For me, as someone with an economics rather than engineering background, the contribution to Open Source projects seemed limited, since “I can’t contribute to fixing or improving stuff”. Other areas as Open Data seem much more approachable for me.

His recommendation was to start with contributing in your area of expertise, optimally in an upcoming hackathon. As an example a designer could contribute with graphics, logos or websites during the hackathon. Similarly, someone with an UX background could help to improve acceptance of users, and someone with a business background could support with figuring out the economic feasibility of the solution for both, customers and providers.

That suggestion seems trivial, but for me that was not so explicitly clear. I realize from my own experience that you most likely can’t excel in all areas that are required to operate a successful Open Source business and outside suggestions are always appreciated. So I think, the next time I might just offer my support, being also open to receive a “no thanks”.

## Great, now what?

If you want to learn more about the study and the talks:

- [View](https://oss-studie.ch) and download the study (German)
- [View](https://www.digitale-nachhaltigkeit.unibe.ch/dienstleistungen/auftragsforschung/oss_studie_2021/index_ger.html) and download all slides and view the talks (German)
- Checkout the Association for the promotion of Open Source Software and Open Standards in Switzerland
- Open Source [activity tracker](https://ossbenchmark.com/) of Swiss companies
- [Directory](https://www.ossdirectory.com/) of Swiss open source solutions

_Unfortunately the event was hosted in German only. Translating the website worked for me partially. Let me know in case you are interested but need help with the language part._

If you’re interested in our projects:

- Checkout our [Github](https://github.com/caos) with the two main projects [ZITADEL](https://github.com/zitadel/zitadel) and [ORBOS](https://github.com/caos/orbos)
- Do you want to share or ask something? [ZITADEL Discussions](https://github.com/zitadel/zitadel/discussions) and [ORBOS Discussions](https://github.com/caos/orbos/discussions)
- Can you [help us get better at engaging](https://github.com/zitadel/zitadel/discussions/1919) our Open Source community?
- Do you have [suggestions on how to make it easier to contribute](https://github.com/zitadel/zitadel/discussions/1920) to our project?
- Want to start contributing? Grab a [“good first issue”](https://github.com/zitadel/zitadel/issues?q=is%3Aissue+is%3Aopen--+label%3A%22good+first+issue%22) and checkout [how to contribute](https://github.com/zitadel/zitadel/blob/main/CONTRIBUTING.md)


What is our motivation for Open Source? Does it bring value to our business? Allow me to share some thoughts on an Open Source study and motivation for Open* with you.

Blog.