Moving the Pager: Lessons from an AI-Assisted Alerting Migration

Youssef Chlih · June 24, 2026 · 9 min read

AIPlatform Engineering

Changing where an alert goes is easy. Preserving who gets paged, and when, across hundreds of repositories is harder. With Opsgenie approaching end of support on 5 April 2027, we needed to migrate our alerting while preserving service ownership, routing, and escalation behaviour. We used a coding agent to help, but its early mistakes showed us what we needed to verify before scaling up.

We selected incident.io for its combination of on-call scheduling, alert routing, incident coordination in Slack, workflow automation, and post-incident follow-up. Usability mattered too: engineers needed to find alerts, understand escalations, and change ownership easily. Its AI-assisted summaries and draft post-mortems were another attraction.

I led the migration from Opsgenie and PagerDuty to incident.io, covering more than 350 repositories, multiple monitoring tools, and integrations configured outside code. In this post, I explain how we used a coding agent to prepare the changes, where it missed things, and how we adjusted the process. We prepared the initial pull requests in two weeks. Deployment and team onboarding took three months, with a dedicated task force coordinating workshops and communication.

#What We Needed to Migrate

At first, the task sounds straightforward: replace the old alert destination with the new one. But an alert also needs a service identity, a team owner, and the right escalation behaviour. Changing the destination should preserve whether it pages someone immediately or follows a working-hours path.

The integrations came in several forms:

• CloudWatch alarms routed through SNS.
• Datadog monitors defined in deployment code.
• New Relic alerts configured in monitoring dashboards.
• Availability checks and custom tools sending alerts over HTTP.
• Systems sending email notifications or using phone call routes.

The deployment code varied too: CloudFormation, Terraform, CDK, Ruby scripts, and other resource-creation code. Some notifications lived entirely outside repositories, in monitoring dashboards or email settings. Searching code could cover only part of the migration.

#A Monitoring Tool That Needed a Release

One example was a homegrown monitoring tool whose configuration lived in code. Changing its alert destination required a code change and a release. What looked like an integration setting became a software delivery task.

Some older integrations needed help from engineers familiar with their configuration. Finding the alert source also meant finding the right people to help change it.

Luckily, we already had a general-purpose incident.io HTTP integration with routing for custom alerts and deployment automation. We expanded its use to cover availability checks, homegrown monitoring tools, and alerts sent directly from application code. The tool could use that integration, so we could focus on changing the sender.

#Agreeing on the Routing Rules

We brought these integrations into incident.io through monitoring integrations, a shared HTTP integration, email alert sources, and call routes. We then needed consistent metadata across those paths.

Our developer Portal provided the service catalog and team ownership information. That was the reference for routing metadata, even when a repository used a different name.

For the repository changes, we established a few specific rules:

• A Datadog monitor’s service: tag must match the catalog service name, represented by metadata.name in catalog-info.yaml.
• Check existing team: tags against catalog ownership.
• Add missing service tags, but do not invent a team tag when one is absent.
• Check service parameters and existing team tags on incident.io SNS integrations against the same catalog data.
• Preserve the intended on-call behaviour.

The last rule needed care. A missing flag can select a default rather than disable paging. Fields such as oncall:true and oncall:false needed review because changing or removing them could change the escalation behaviour.

HTTP and email senders also needed attention. A destination URL or email address says where to deliver the alert, but does not necessarily identify its service or owner. We had to check the metadata available in each payload or message.

#Using a Coding Agent

The agent helped us find old integrations, classify deployment patterns, and prepare migration code changes. It also helped inspect stale or conflicting PRs, verify tags, and exclude archived repositories from the active migration scope.

#What the First Pass Missed

The agent initially focused on familiar patterns: monitors as code and CloudWatch alarms in infrastructure as code. It missed some custom alert senders that used API keys because they did not look like the monitoring resources it was targeting.

Reviewing a few generated PRs also exposed missing tags and incorrect tags copied from code. A name in a deployment file could look reasonable while differing from the service or team recorded in the Portal.

Those reviews helped us establish a baseline across CloudFormation, Terraform, CDK, Ruby deployment scripts, and other resource-creation code. We needed the same checks regardless of the language or framework.

#Encoding the Baseline in a Skill

We put that baseline into a reusable coding-agent skill. It described where to look, which integration patterns to recognise, what to change, and how to verify service and team tags. The skill used service-catalog metadata from the Portal for those checks.

The workflow became:

1. Find old alerting integrations and identify their deployment pattern.
2. Prepare the code changes using the agreed migration rules.
3. Open a PR for review.
4. Address CI failures, conflicts, and ownership mismatches.
5. Run a follow-up validation pass on migrated repositories.

As we refined the skill, we ran it again and updated PRs to check their tags against Portal data. This was useful even when a repository already used the new endpoint: the destination could be correct while its routing metadata still needed work.

#An Example: Service Name Versus Metric Name

Here is a simplified example:

Tags:
  - team:old-team-name
  - !Sub 'service:${Service}'

The monitor has both tags, but we still need to check their values. A deployment parameter can be transformed for use as a metric namespace. For example, the metric name might be example_service, while the catalog service name is example-service.

Using the metric name for alert routing can produce a mismatch. The correction keeps the metric namespace for queries and uses the catalog identity for the alert tag:

Tags:
  - team:example-team
  - service:example-service

This is a small change, but the values determine which service and team the alert is associated with. Checking that a tag exists is only the first step.

#Reviewing the Changes and Testing Delivery

Some decisions stayed with people:

• Whether a team wanted dual delivery during the transition.
• How custom HTTP and email alerts should map to services and escalation paths.
• Whether an alert should page the on-call engineer.
• Whether a CI failure came from the migration or an unrelated dependency.

The agent could prepare a change and flag an inconsistency. The owning team still needed to review its operational meaning and test the integration.

#A Catch-All Channel for Routing Exceptions

We set up a catch-all channel for alerts that could not be routed because their service or team tags were missing or incorrect. We monitored it during the migration to make routing exceptions visible for review.

We asked teams to send test alerts and check whether they were paged. If they were not, we asked them to check the catch-all channel to see whether the alert had ended up there. This helped them identify routing issues during testing.

Repository review and incoming-alert checks served different purposes. A PR showed what we intended to change; the catch-all channel helped us spot routing gaps in alerts arriving at the new platform.

#Two Weeks to Prepare, Three Months to Roll Out

The initial batch of PRs took two weeks to prepare. Deployment and onboarding ran over three months, planned around each team’s existing commitments and capacity.

We set up a dedicated task force to coordinate communication and workshops with the teams owning the repositories and services. Teams needed time to review changes, test integrations, and prepare to operate the new alerting configuration.

By the end of that period, we had onboarded most teams with monitors as code, along with teams whose New Relic alerts were configured manually. Preparing the code changes was one milestone; having the team ready to operate the new setup was another.

Across the migration and subsequent validation work, more than 500 PRs were created in total. We migrated more than 350 repositories, each with roughly 5-10 alerts, alarms, or monitors.

The PR totals include later migration and tag-validation work. They describe the accumulated work, rather than the initial two-week batch. Some changes were implemented through follow-up PRs that replaced earlier attempts. Dashboard-configured integrations needed separate checks.

#What Improved for Engineers

The new interface made it easier to filter alerts, inspect escalations, reassign an alert to another person, and change the responsible team. Requesting cover or swapping an on-call shift also became easier.

Team names were current and searchable in the Portal, giving engineers a consistent reference when an alert used a historical name. Team configuration was visible in code, so engineers could inspect it and propose changes through PRs.

These were useful improvements for the people handling alerts, beyond the work of changing integration destinations.

#What I Would Reuse Next Time

Several parts of the approach are useful for other migrations:

• Review a few early code changes before scaling up. Use what you find to define a baseline across deployment patterns.
• Give the agent a reference for metadata. Validate identity and ownership against a catalog instead of inferring them from code.
• Include custom senders in the inventory. Check monitoring dashboards, scripts, HTTP integrations, email settings, and call routes as well as infrastructure code.
• Check test alerts that do not page anyone. Ask teams to look in the catch-all channel for alerts whose routing needs attention.
• Plan the team rollout alongside the code changes. Workshops and testing take time even when the code changes are ready.

For the next migration, I would also introduce progress tracking and metadata checks earlier. CI checks for service tags and existing team ownership, plus a standard helper for SNS integrations, could catch some of the follow-up work before a PR reaches review.

#Conclusion

The coding agent helped us prepare and revise changes across more than 350 repositories. Its early mistakes gave us concrete checks to add to the process: look beyond familiar monitoring resources, validate service identity against the catalog, and review the fields that control paging. Encoding those checks in a skill made later changes more consistent.

Preparing the initial PRs in two weeks did not remove the work of rolling them out. Teams still needed to review ownership, test delivery, and become comfortable operating the new setup. The catch-all channel helped us spot routing exceptions as alerts arrived, while workshops and coordination supported the three-month rollout.

If you are planning a similar migration, start with a few representative integrations. Review the changes the agent proposes to your alerting configuration, and use those findings to refine its instructions. Validate service identity and ownership against your catalog, then test that alerts reach the right people with the intended escalation behaviour before scaling the rollout to more teams.