AI code review on Azure DevOps: Services vs Server, per tool

Posted on Nov 4 • Originally published at mergerequests.dev

Azure Repos ships in two editions, and the AI review options are not the same in both. Microsoft's Copilot code review is documented for Azure DevOps Services and carries a limited preview label. Azure DevOps Server has no Microsoft-native reviewer at all, so a Server team runs PR-Agent from a pipeline, points a vendor webhook at Service Hooks, or self-hosts something. Which edition you are on decides the setup work, the permissions you need, and where the reviewer reads your coding standards from.

That last part is where most rollouts go sideways, so it is worth walking through the routes one at a time.

The get started page is headed Azure DevOps Services, and the feature is in limited preview: no service level agreement, and functionality can change or be removed. Prerequisites are an Azure DevOps organization, a Git repository in Azure Repos (TFVC is not supported), and an Azure subscription linked to the organization, because usage bills through Azure Cost Management instead of through seats.

Enablement cascades across scopes. A Project Collection Administrator turns the feature on at the organization level and can enable it for every project at once. A Project Administrator scopes it to specific projects, and a repository owner enables it per repository when project settings allow overrides. Individual users opt in through Preview features unless an administrator has already enabled the preview for everyone.

Automatic reviews are a separate switch from enablement. Copilot reviews pull requests on its own only where you add a branch policy on the target branch. Without that policy, someone has to add Copilot as a reviewer by hand on each pull request, which is fine for a pilot and not fine for a team. The default review effort is set at project or repository level, and higher effort consumes more tokens, so the effort setting is also a cost setting.

For a Server install, none of this exists.

Copilot takes instructions at the organization and project level through Azure DevOps settings, and at the repository level through a file. The instructions page accepts two paths for that file:.github/copilot-instructions.md or.azuredevops/copilot-instructions.md, both at the repository root. Path-scoped files go in.github/instructions/ or.azuredevops/instructions/ as *.instructions.md, with YAML front matter carrying an applyTo value.

A path-scoped file without applyTo is skipped. Nothing errors, the file is committed, and reviews keep running without it. If you have a rules file that seems to have no effect, that front matter value is the first thing to check.

The catch that changes how you roll standards out is which branch the instructions come from. Repository-level and path-scoped files are read from the pull request's target branch only. A pull request that adds or edits the instructions file is not reviewed against the new text. Those rules apply after the change merges into the target branch. If you are introducing a standard, land the file on the default branch first, then let the later pull requests pick it up.

When instructions exist at several scopes, Copilot uses all applicable ones during a review. The docs do not promise a precedence order for an organization rule that contradicts a repository rule, so that is a conflict to avoid rather than resolve.

On the server edition the choice is between self-hosted tools and hosted vendors that reach into your instance.

PR-Agent lists Azure DevOps as a supported git provider, and the support matrix lists an Azure DevOps pipeline as supported usage. A Server team can run it as a pipeline job or behind a webhook. It is community-owned and MIT licensed, review categories and prompts live in configuration files you own, and the matrix lists repo context files such as AGENTS.md. One caveat to note before you plan around its help tooling: /help_docs has been disabled since v0.36.1 pending a fix for a credential-exposure issue.

Open Code Review from Alibaba is a different shape. It installs from npm as ocr, works on a git diff, a commit range, or a whole-file scan, and needs an LLM endpoint rather than a forge integration. On a firewalled Server install that is convenient, and it also means the tool does not post comments to a pull request by itself unless you wire that up. Its published benchmark is the project's own, built from 50 repositories and 200 pull requests with engineer-annotated issues, and it reports higher precision than a general-purpose agent on the same model at roughly a ninth of the tokens, with lower recall as a stated trade-off. Treat it as the project's number, not a third-party measurement.

Kodus connects to Azure DevOps with a Personal Access Token and documents no OAuth path for that provider, unlike GitHub and GitLab. The PAT scopes the docs list are Analytics Read, Code Read & Write, Graph Read, Identities and Groups Read, Project and Team Read, and User Profile Read, with Code write access being the one that lets it post comments. If the connection succeeds but nothing appears on new pull requests, the PAT scopes are the first thing the troubleshooting page points at.

One Azure-specific detail matters if you self-host it: the Azure Repos webhook URL has to carry an encrypted token query parameter, derived by encrypting CODE_MANAGEMENT_WEBHOOK_TOKEN with CODE_MANAGEMENT_SECRET using AES-256-CBC. Requests without it get a 403. The webhook page also notes the URL has to reach the webhooks service on port 3332, or your reverse proxy has to route the webhook path there. And on rules, Kody reads coding-agent instruction files including.github/copilot-instructions.md and.github/instructions/**/*.instructions.md; the.azuredevops/ paths are not in that list, so a team standardizing on the Azure-native path should know which tools will read it.

Less clearly than you would want, and the difference matters.

CodeRabbit's Azure DevOps page covers a PAT or a Microsoft Entra service principal, Project Administrator or Project Collection Administrator rights, and the Edit Subscriptions permission on Service Hooks so the webhook can be installed. It notes that permission cannot currently be granted through the UI and that it is granted per project, which becomes real work across a large organization. The page does not split Azure DevOps Services from Azure DevOps Server, so it does not establish Server support on its own.

Qodo's on-premises Azure DevOps guide walks through a Microsoft Entra app registration, a service account added to the organization with Basic access and Project Administrators, and a Service Hooks webhook subscribed to PR Created, PR Updated, PR Commented and PR Merged. Those steps assume dev.azure.com and Entra ID, which are Azure DevOps Services constructs. The deployment matrix separately marks on-prem as unsupported for Azure DevOps Cloud. Read together, the guide covers running Qodo on-premises against Azure DevOps Services. It does not document an Azure DevOps Server install.

Every cell reflects the vendor or project page linked above. Where a cell says not documented or not stated, the page I read does not cover that edition, and that gap is the thing to verify before you commit. The wider edition coverage comparison across all three forges covers the same question for GitLab Self-Managed and Bitbucket Data Center.

On Azure DevOps Services, Copilot code review is the shortest path to a first automated review. An administrator turns it on at the organization level, billing runs through a subscription you already have, and reviewers appear on pull requests without a webhook to maintain. What you accept in return is the preview status, the missing precedence order when instructions conflict, and the target-branch rule for instruction files. If your standard is that a reviewer must apply a rules file changed inside the same pull request, Copilot will not do that, and you want a tool that reads the rule at review time.

On Azure DevOps Server the choice is between PR-Agent and a self-hosted product. PR-Agent is the lower-commitment option: MIT, no vendor account, configuration stored in files in your repository, and a documented pipeline trigger. What you pay for it is operational work, since you own the pipeline, the model key and the webhook endpoint. A UI for managing rules, or metrics on what the reviewer found over time, is where a product like Kodus starts to make sense, and the price of that is running the service yourself, including the webhook token and the port exposure above.

The cheapest test before committing to any of them is one pull request. Put a single rule in the file that route reads, open a pull request that breaks the rule, and see which reviewers flag it. On Azure Repos, the file that reaches the most reviewers is.github/copilot-instructions.md, because Copilot reads it and Kody detects it among its supported patterns. If your repository already speaks.azuredevops/, keep both paths in sync or the tools will end up reviewing against different standards.

Is Copilot code review available on Azure DevOps Server?

No. Microsoft documents it for Azure DevOps Services, and the documentation is headed for that edition. Server teams run a self-hosted reviewer such as PR-Agent, or wire a vendor integration into Service Hooks.

Where does Copilot code review read custom instructions from?

Organization and project settings, repository files at.github/copilot-instructions.md or.azuredevops/copilot-instructions.md, and path-scoped *.instructions.md files carrying an applyTo value. Repository-level and path-scoped files are read from the pull request's target branch, so a change to them applies after it merges.

Can CodeRabbit or Qodo review Azure DevOps Server pull requests?

Neither vendor page establishes Server support. CodeRabbit documents an Azure DevOps integration through a PAT or an Entra service principal, and Qodo's on-premises guide uses dev.azure.com and Entra ID, which are Azure DevOps Services constructs.