Algorithms, Audits, and the Auditor

Remarks as prepared for delivery

Today, the Board considers a staff recommendation, Proposed Amendments Related to Aspects of Designing and Performing Audit Procedures that Involve Technology-Assisted Analysis of Information in Electronic Form, to improve PCAOB audit standards concerning the audit profession’s increasing reliance on technology-based tools, including “data analysis” or “data analytics.”

As we all know, advances in technology are transforming the way we work, and the audit industry is not isolated from these changes. With an ever-increasing volume of data, from an expanding range of data sources, auditors are facing new challenges as they seek to gather sufficient, and appropriate, evidence.

The global pandemic accelerated the adoption of data analytics, distributed ledger technology (DLT), robotic process automation (RPA), machine learning (ML), and many other emerging technologies.

Regardless of the form of the technology, all of them include a step-by-step plan called an algorithm. Algorithms can be complex but start simply. A restaurant chef follows an algorithm, or a step-by-step process, when making a sauce. The quality of the sauce is dependent on a variety of factors, including the quality of the inputs – the ingredients.

Last November, OpenAI’s text-writing tool, ChatGPT, stunned the world with its leaps in understanding, analyzing, and generation of human-like language (i.e., “generative artificial intelligence (AI)”). Some audit firms recently have announced their intentions to invest and deploy this generative AI technology in audits.

The opportunities this technology provides for an increase in the coverage of the audit, a decrease in effort, and potentially deeper insights should translate into improvements in risk assessments and audit efficiency that can be a win-win for everyone.

There are significant challenges, however, that may be often overlooked.

For example, I asked ChatGPT to draft a statement for today’s open meeting. And the statement it produced was a good one – extolling the virtues of new technology and improvements in the audit.

ChatGPT’s statement failed, however, to include anything about potential problems, such as how our inherent perceptions about technology can lead to bias.

As one recent research study observed, our “personal beliefs about new technology — that it’s wondrous, complex, and alien — prompt specific, unconscious biases about how and why it’s better than older options.”1 That means that we see what we want to see in the output.2

Moreover, while two auditors may input the exact same information into ChatGPT, the output may not be the same. And sometimes, that output may be flat out wrong.

The output from ChatGPT and other so-called AI is, at best, a probabilistic prediction based on data correlation inferences. Essentially, it’s a really good guess, but that’s all it is. It is not an absolute fact.

This may be the most significant challenge for auditors – objectively reviewing the output and concluding that the information produced is relevant, reliable, and achieves the audit objective, and therefore, constitutes audit evidence. The actual techniques or tools used for technology assisted analysis do not constitute audit evidence.

Today’s amendments, combined with the Board’s previous proposals, such as an audit firm’s system of quality control,3 are intended to begin our process of addressing the impact of technology in the audit.

The amendments should support innovation in the audit and address where the PCAOB has observed lapses or weaknesses in the relevance and reliability of the results derived from technology-assisted analysis.

The amendments emphasize the need for auditors to understand the company, appropriately assess the risks of material misstatement of the financial statements and develop and deploy an appropriate audit response.

Accordingly, auditors should understand both the inputs and outputs when deploying technology-assisted data analysis before forming audit conclusions.

Technology or computer-assisted analysis is a tool that can enhance, but cannot replace, professional skepticism and professional judgment. Consequently, the foundational principles for the audit remain paramount, which is why the Board has not elevated technology-assisted analysis to an audit procedure.4

The proposal we are considering is a targeted amendment of the Board’s standard on audit evidence. More work remains, however, to create a comprehensive framework for appropriately utilizing technology during the audit but not becoming overly reliant on it to the degree that it becomes an actual impediment to audit quality.

We are aided in that objective by my fellow Board member, Christina Ho, who leads the Technology Innovation Alliance Working Group (TIA Working Group). The TIA Working Group advises the Board on the use of emerging technologies by auditors and financial statement preparers and its potential impact on audit quality. The Group offers recommendations on how the Board’s existing or future oversight programs might address the use of emerging technologies by auditors.

In reviewing today’s proposal, I hope that commenters will in provide answers to all the queries in the release, including questions about whether the requirements are fit-for-purpose and appropriately tailored to situations when “auditors strategically leverage data analytics to provide clients with business-related insights.”5

I am particularly interested in answers to the following questions:6

  • Are there important conflicts that we should address when auditors use audit procedures for more than one purpose?
    • Is auditor independence protected?
    • Are there threats to audit quality?
  • Is there unnecessary confusion in the use of the terms “data analysis” or “data analytics,” which are broadly used, but are neither “analytical procedures” nor are they “substantive analytical procedures”?

We would not be here today if not for the hard work and dedication of Donna Silknitter, who has tirelessly advanced this project over the last year.

I also want to thank Robert Kol from the Office of the Chief Auditor, as well as Nicholas Galunic from the Office of Economic Analysis for their contributions to today’s recommendation.

Today’s recommendation was also informed by the PCAOB’s former Data and Technology Task Force,7 led by Nicholas Grillo, from the Office of the Chief Auditor. The Task Force members helped the staff by providing their insights into the use of technology-based tools by auditors and preparers, and I thank them for their time and contributions.

Thank you, and I look forward to reading your comments.

1 Kimberly D. Elsbach and Ileana Stigliani, “Evaluating New Technology? You’re More Biased Than You May Realize: Unconscious ideas about new technology can lead to poor investment decisions,” MIT Sloan Management Review, Sep 23, 2020. Evaluating New Technology? You’re More Biased Than You May Realize (

2 Auditors are required to be both independent in fact – that is, objective and unbiased in attitude – and independent in appearance to others. An auditor’s exercise of professional skepticism considers the impact of management bias and the auditor’s own bias that could affect the auditor’s own judgments. Auditors may have biases related to electronic information. For example, a tendency to favor output generated from automated systems, even when contradictory information raises questions as to whether such output is reliable, illustrates a form of bias. Exercising professional skepticism, including critically assessing information related to the audit, helps the auditor address the effects of potential bias on professional judgment and decision-making. See PCAOB Release No. 2023-001; see also Raymond S. Nickerson, Confirmation Bias: A Ubiquitous Phenomenon in Many Guises, 2 Review of General Psychology 175 (1998). 48

3  Proposed QC 1000 would require audit firms to ensure, among other things, that technological resources (1) “have the capacity, integrity, resiliency, availability, reliability, and security necessary” and (2) “are obtained or developed, implemented, maintained, and used” appropriately. See PCAOB Release No. 2022-006 November 18, 2022.

4 Audit procedures can be classified into either risk assessment procedures or further audit procedures (tests of controls and substantive procedures) and include inspection, observation, inquiry, confirmation, recalculation, reperformance, and analytical procedures. See PCAOB AS 1105, Audit Evidence.

5 Ashley A. Austin, Tina D. Carpenter, Margaret H. Christ, and Christy S. Nielson, The Data Analytics Journey: Interactions Among Auditors, Managers, Regulation, and Technology, 38 Contemporary Accounting Research 1888 (2021) ,“[A]uditors report that they strategically leverage data analytics to provide clients with business-related insights. However, regulators voice concerns that this practice might impair auditor independence and reduce audit quality.”

6 See Release questions: “3. What other requirements may need to be included in PCAOB standards to address the use of technology-assisted analysis in audits?”: “6. Are the proposed requirements that specify the auditor’s responsibilities when using audit evidence from an audit procedure to achieve more than one purpose clear and appropriate?”;” 12. Are the proposed amendments that update certain terminology in AS 1105 clear and appropriate? If not, what changes should be made?”

7 The PCAOB established the Data and Technology Task Force in 2017 to help a staff research team obtain insights into the use of data analytics and certain emerging technologies. The Board allowed the task force to expire on May 16, 2022. The members included, Helen L. Brown-Liburd, Associate Professor, Rutgers Business School, Rutgers, The State University of New Jersey; Brian P. Collins, Partner, Baker Tilly Virchow Krause, LLP; Mary Grace Davenport, Partner, PricewaterhouseCoopers LLP; Jason Guthrie, Managing Director, Ernst & Young LLP; Robert H. Herz, Chief Executive Officer, Robert H. Herz LLC and Executive-in-Residence, Columbia Business School, Columbia University; Douglas L. Maine, Limited Partner and Senior Advisor, Brown Brothers Harriman & Co.; Jeffrey D. Nuechterlein, Managing Partner, Nue Capital LLC; Nicole Oberst, Managing Director, Deloitte & Touche LLP; Sandra J. Peters, Head of Financial Reporting Policy, CFA Institute; D. Scott Showalter, Professor of Practice, Department of Accounting, Poole College of Management, North Carolina State University; Ian Wildenborg, Partner, KPMG LLP and Brian Wolohan, National Partner, Grant Thornton LLP