When corporate user directories become too large that they need server-side filtering
Large corporations often deal with an impressive amount of entities in their IdPs. And that volume doesn’t reflect just the armies of users. In fact, it’s not uncommon for these companies to manage hundreds of thousands of user groups.
When such gigantic user directories are sent into Jira or Confluence Data Center, the sync can run for hours, and take significant network resources that may compromise performance.
Additionally, such lengthy syncs can be difficult to schedule, monitor, and troubleshoot — particularly if you want to maintain daily accuracy of all user accounts.
But the worst part of it is that more often than not, only a subset of users is really needed in the Atlassian application, making all that sync effort superfluous.
A strategic commitment with our largest customers
We decided to create server-side filtering as an in-built option for those customers who only need to sync a small subset of their user directory.
The motivation to address this issue is embedded in our strategy: to make SAML SSO and User Sync fit for any scale, no matter how big. And we’re happy and proud to say that we’ve solved the challenge!
Some of the folks that we have active support cases with have already tested the new capabilities and given us their feedback: their updates have gone from hours to minutes:
Hey, I’ve tested the new Server Side Filtering. Smooth like a charm with our 12k groups, in total it was about 20x faster than last time.
Thanks for suggesting that I filter directly from Azure. Syncs take half the time now!
Be it 2,000% or 50% faster, if the number of user groups in the IdP is an issue, you now have the means to address it!
Lightning-Fast User Synchronizations with SAML SSO and User Sync
Faster synchronizations are not magic: they are more efficient because they only work on relevant data.
Instead of filtering groups in the Atlassian instance, the server-side filtering asks the IdP to only send what’s needed. The filtering is processed internally by the IdP and doesn’t have to be transmitted via the API.
The old method: filtering groups locally
Up until now, when an Azure connector synced groups, this is in simplified terms what was going on in the background whenever our customers configured required groups as a condition in the connector:
- All groups were sent from the IdP
- In order to obtain the required groups, a regular expression or a groovy script was applied to the full list of groups fetched from the IdP . For example, every group starting with “demo”.
- Finally, the connector would make additional calls to sync the user pertaining to the resulting required groups.
When groups are in the thousands, the first step becomes very long. And you haven’t even started updating or creating users by then.
Server-Side Filtering – or filtering done right on the IdP
To alleviate this, we’ve added a server-side filter feature. This means that while the sync process is triggered in Jira or Confluence, the first filter happens in the IdP.
Find our comprehensive walkthrough here.
Server-Side Filtering with Azure AD (Microsoft Entra)
When building Server-Side Filtering queries for Azure AD, our customers can employ the use of commands like ‘StartsWith’ or ‘Eq – Equal’ to filter groups with specific starting names or matching exact strings.
Find more information in the Microsoft Graph API documentation
Server-Side Filtering with Okta
In the case of Okta, there are very similar options available to find group names by exact names or starting with a string, to find groups by exact ID, or even to filter using multiple expressions. You can access the Groups Endpoint Documentation for more details.
An additional benefit of Server-Side Filtering when working with Okta is that the more you filter within the IdP, the less of a toll user synchronizations will have on your token quota!
When should I use Server-Side Filtering?
We recommend using Server-Side Filtering if your Azure AD or Okta instance contain 1,000 groups or more. Depending on the complexity and other requirements, it might interesting to sometimes start even lower, at about 500 groups.
Additionally, Okta has a strict rate limiting policy, so it’s recommendable to minimize the number of entities that it sends.
And here’s an important warning from our product owner: because of how different naming conventions can be, in many setups it won’t be able to get down to the handful of groups that you actually need in the Atlassian application. This means that you cut down to get a set of groups that’s almost there, and you refine the filter locally. First step is for efficiency, second step is for accuracy.
What are the steps to enable Server-Side filtering?
The new feature can be found in the Group Management settings of Azure AD (Entra) and Okta.
When activating the option to set “Required groups”, simply add the desired query in the displayed editor, as you can see in this example
When should I use Local Filtering?
It’s still recommendable to use the existing option to filter groups locally. Particularly, in installations hosting 500 groups or less in their IdPs should experience sync runs with a duration of just some seconds. Seems acceptable!
Remember that local filtering is way more flexible, since it can use regex expressions as opposed to the operations “startswith” and “equal”. And you’re very likely to use local filtering also once you get the desired subset of groups from the IdP to grab exactly the required groups that those filters won’t serve you out of the box!
What are the steps to enable Local filtering?
Local filtering can be found next to the new feature, in the Group Management settings of Azure AD (Entra) and Okta.
When activating the option to set “Required groups”, simply choose whether you will determine the required groups with a Regular Expression, or using a Groovy Script.
Dual filtering power: When Local and Server-Side methods work together
Yes, of course! What’s more: local filtering and server-side filtering are not mutually exclusive! Both aspects can be combined to find the right granularity for your exact needs. For example, you could:
- Filter groups server-side to get only groups containing “jira”. Imagine that this operation results in 16 groups.
- Define a list or a regular expression locally to determine the subset of the 8 groups you are actually interested in.
Are there any limitations?
As I’ve written above, we currently support Local and Server-Side Filtering for Azure AD (Microsoft Entra ID) and Okta. Our team is working on supporting additional IdP connectors soon.
Besides that, the limitations are the ones implemented by the IdPs themselves:
- Azure AD allows a maximum of 15 conditions in their queries.
- Okta, as we have mentioned, has a rate limit that, once exceeded, will return a 429 error status for each subsequent API call.
In opting for either Server-Side Filtering or Local Filtering, consider these variables to leverage the full capabilities of our SAML SSO apps to your advantage.
Conclusion
In conclusion, our new filtering options are geared towards providing the maximum efficiency to our largest customers, when they only need to provision a subset of their users in their Atlassian products. Working around rate limits, minimizing the duration of user synchronizations, and ensuring performance of your critical Atlassian applications is now so much easier.
Give it a try or contact us to help you set up your perfect server-side query
