Understanding Playwright's Accessibility Tree: Providing Better Context for AI-Driven Locator Generation

#testing #automation #playwright #ai #agenticai

In the first article of this series, we explored why generic locator generation can become challenging when automating complex enterprise applications such as Salesforce Lightning.

We discussed how multiple elements can share identical accessible names and why identifying an element without understanding its surrounding context can lead to unreliable automation.

What if we could provide AI with a structured representation of the page instead of asking it to identify elements from an entire page?

One approach worth exploring is Playwright's Accessibility Tree and its accessibility snapshot capabilities.

A browser does not simply render HTML elements visually. It also exposes information about the interface to assistive technologies such as screen readers.

This information is represented through the Accessibility Tree.

Unlike the DOM, which describes the document structure, the Accessibility Tree focuses on how elements are presented and understood by assistive technologies.

For example, consider the following HTML:

<button aria-label= "Edit customer record" > <svg> <!-- Edit icon --> </svg> </button> Enter fullscreen mode Exit fullscreen mode The DOM contains a button and an SVG element.

However, its accessibility representation can expose information such as:

This semantic information is particularly useful when working with Playwright.

Instead of depending entirely on CSS selectors or DOM attributes, we can use accessible roles and names to identify elements.

const editButton = page. getByRole ( ' button ', { name: ' Edit customer record ' }); await editButton. click (); Enter fullscreen mode Exit fullscreen mode This is one of the reasons Playwright recommends user-facing locators such as roles and labels.

Playwright provides the locator.ariaSnapshot() API to retrieve an accessibility-oriented representation of a locator's contents.

const snapshot = await page. locator ( ' body ' ). ariaSnapshot (); console. log ( snapshot ); Enter fullscreen mode Exit fullscreen mode The output is a YAML-like representation of accessible elements.

For example, a simplified representation might look like this:

- main: - heading "Customer Cases" [level=1] - region "Parent Case": - heading "Case #1001" - button "Edit" - region "Child Case": - heading "Case #1002" - button "Edit" Enter fullscreen mode Exit fullscreen mode The example above is illustrative. Actual output depends on the application's accessible markup and browser state.

Instead of receiving only a collection of buttons named Edit, we can potentially understand the sections in which those buttons appear.

This additional information can help distinguish between otherwise similar elements.

Consider a Salesforce workflow involving parent and child cases.

const editButton = page. getByRole ( ' button ', { name: ' Edit ' }); Enter fullscreen mode Exit fullscreen mode If multiple buttons match, Playwright's strictness checks prevent the automation from arbitrarily selecting one.

const editButton = page. getByRole ( ' button ', { name: ' Edit ' }). nth ( 0 ); Enter fullscreen mode Exit fullscreen mode However, this introduces a dependency on element ordering.

A more context-aware approach would be to identify the relevant section first.

const parentCase = page. getByRole ( ' region ', { name: ' Parent Case ' }); const editButton = parentCase. getByRole ( ' button ', { name: ' Edit ', exact: true }); await editButton. click (); Enter fullscreen mode Exit fullscreen mode This approach limits the search to the intended region.

The important distinction is that we are no longer identifying the button independently. We are identifying it within the context of the parent case.

Of course, this assumes the application exposes a meaningful accessible region with the appropriate name. Real Salesforce markup may require a different scoping strategy.

Now let's introduce AI into the workflow.

Imagine an AI agent receiving the following instruction:

"Open the parent case and change its status to In Progress."

If we provide only a large collection of page elements, the model needs to infer these relationships from potentially ambiguous information.

However, if we provide a structured accessibility snapshot, the model gets additional semantic information.

A simplified instruction could look like this:

Task: Update the Parent Case status to In Progress. Relevant page structure: Region: Parent Case Heading: Case #1001 Button: Edit Region: Child Case Heading: Case #1002 Button: Edit Instruction: Identify the Edit button within the Parent Case region. Do not interact with the Child Case. Enter fullscreen mode Exit fullscreen mode This is not a complete AI automation implementation. It illustrates how structured context and explicit instructions can help reduce ambiguity during locator identification.

There is an important limitation worth understanding.

The Accessibility Tree does not necessarily expose every element present in the DOM.

Some elements may be excluded because they are not accessible, are hidden, or do not contribute meaningful accessibility information.

Additionally, Salesforce Lightning contains dynamically rendered components, custom controls and complex interactions.

An accessibility snapshot may not always provide enough information to identify every required element.

Therefore, accessibility snapshots should not be treated as a replacement for the DOM or as a guaranteed solution to locator ambiguity.

Instead, they can serve as another source of structured information for an automation agent.

Playwright already provides powerful locator strategies.

The challenge in AI-driven automation is not simply generating a valid locator. It is generating a locator that identifies the correct element within the intended business context.

Accessibility information provides a useful foundation because it exposes semantic information such as roles, accessible names and relationships between interface elements.

When combined with contextual instructions and scoped locator strategies, this information can help an AI agent make more informed locator decisions.

However, there is still an important question:

Can we make AI identify the relevant section first, and then generate locators only within that section?

In Part 3, I'll explore a context-aware locator generation approach using Playwright, TypeScript and AI.

We'll look at how an automation agent can:

The goal is to move beyond generic locator generation towards context-aware AI-driven browser automation.

Part 3: Building a Context-Aware AI Locator Generation Approach with Playwright.