Algorithms, Audits, and the Auditor

Remarks as prepared for delivery

Today, the Board considers a staff recommendation, Proposed Amendments Related to Aspects of Designing and Performing Audit Procedures that Involve Technology-Assisted Analysis of Information in Electronic Form, to improve PCAOB audit standards concerning the audit profession’s increasing reliance on technology-based tools, including “data analysis” or “data analytics.”

As we all know, advances in technology are transforming the way we work, and the audit industry is not isolated from these changes. With an ever-increasing volume of data, from an expanding range of data sources, auditors are facing new challenges as they seek to gather sufficient, and appropriate, evidence.

The global pandemic accelerated the adoption of data analytics, distributed ledger technology (DLT), robotic process automation (RPA), machine learning (ML), and many other emerging technologies.

Regardless of the form of the technology, all of them include a step-by-step plan called an algorithm. Algorithms can be complex but start simply. A restaurant chef follows an algorithm, or a step-by-step process, when making a sauce. The quality of the sauce is dependent on a variety of factors, including the quality of the inputs – the ingredients.

Last November, OpenAI’s text-writing tool, ChatGPT, stunned the world with its leaps in understanding, analyzing, and generation of human-like language (i.e., “generative artificial intelligence (AI)”). Some audit firms recently have announced their intentions to invest and deploy this generative AI technology in audits.

The opportunities this technology provides for an increase in the coverage of the audit, a decrease in effort, and potentially deeper insights should translate into improvements in risk assessments and audit efficiency that can be a win-win for everyone.

There are significant challenges, however, that may be often overlooked.

For example, I asked ChatGPT to draft a statement for today’s open meeting. And the statement it produced was a good one – extolling the virtues of new technology and improvements in the audit.

ChatGPT’s statement failed, however, to include anything about potential problems, such as how our inherent perceptions about technology can lead to bias.

As one recent research study observed, our “personal beliefs about new technology — that it’s wondrous, complex, and alien — prompt specific, unconscious biases about how and why it’s better than older options.”¹ That means that we see what we want to see in the output.²

Moreover, while two auditors may input the exact same information into ChatGPT, the output may not be the same. And sometimes, that output may be flat out wrong.

The output from ChatGPT and other so-called AI is, at best, a probabilistic prediction based on data correlation inferences. Essentially, it’s a really good guess, but that’s all it is. It is not an absolute fact.

This may be the most significant challenge for auditors – objectively reviewing the output and concluding that the information produced is relevant, reliable, and achieves the audit objective, and therefore, constitutes audit evidence. The actual techniques or tools used for technology assisted analysis do not constitute audit evidence.

Today’s amendments, combined with the Board’s previous proposals, such as an audit firm’s system of quality control,³ are intended to begin our process of addressing the impact of technology in the audit.

The amendments should support innovation in the audit and address where the PCAOB has observed lapses or weaknesses in the relevance and reliability of the results derived from technology-assisted analysis.

The amendments emphasize the need for auditors to understand the company, appropriately assess the risks of material misstatement of the financial statements and develop and deploy an appropriate audit response.

Accordingly, auditors should understand both the inputs and outputs when deploying technology-assisted data analysis before forming audit conclusions.

Technology or computer-assisted analysis is a tool that can enhance, but cannot replace, professional skepticism and professional judgment. Consequently, the foundational principles for the audit remain paramount, which is why the Board has not elevated technology-assisted analysis to an audit procedure.⁴

The proposal we are considering is a targeted amendment of the Board’s standard on audit evidence. More work remains, however, to create a comprehensive framework for appropriately utilizing technology during the audit but not becoming overly reliant on it to the degree that it becomes an actual impediment to audit quality.

We are aided in that objective by my fellow Board member, Christina Ho, who leads the Technology Innovation Alliance Working Group (TIA Working Group). The TIA Working Group advises the Board on the use of emerging technologies by auditors and financial statement preparers and its potential impact on audit quality. The Group offers recommendations on how the Board’s existing or future oversight programs might address the use of emerging technologies by auditors.

In reviewing today’s proposal, I hope that commenters will in provide answers to all the queries in the release, including questions about whether the requirements are fit-for-purpose and appropriately tailored to situations when “auditors strategically leverage data analytics to provide clients with business-related insights.”⁵

I am particularly interested in answers to the following questions:⁶

• Are there important conflicts that we should address when auditors use audit procedures for more than one purpose?
  • Is auditor independence protected?
  • Are there threats to audit quality?
    
• Is there unnecessary confusion in the use of the terms “data analysis” or “data analytics,” which are broadly used, but are neither “analytical procedures” nor are they “substantive analytical procedures”?
  

We would not be here today if not for the hard work and dedication of Donna Silknitter, who has tirelessly advanced this project over the last year.

I also want to thank Robert Kol from the Office of the Chief Auditor, as well as Nicholas Galunic from the Office of Economic Analysis for their contributions to today’s recommendation.

Today’s recommendation was also informed by the PCAOB’s former Data and Technology Task Force,⁷ led by Nicholas Grillo, from the Office of the Chief Auditor. The Task Force members helped the staff by providing their insights into the use of technology-based tools by auditors and preparers, and I thank them for their time and contributions.

Thank you, and I look forward to reading your comments.