How to Avoid the Ethical Nightmares of Emerging Technology

Facebook, which was created in 2004, amassed 100 million users in just four and a half years. The speed and scale of its growth was unprecedented. Before anyone had a chance to understand the problems the social media network could cause, it had grown into an entrenched behemoth.

In 2015, the platform’s role in violating citizens’ privacy and its potential for political manipulation was exposed by the Cambridge Analytica scandal. Around the same time, in Myanmar, the social network amplified disinformation and calls for violence against the Rohingya, an ethnic minority in the country, which culminated in a genocide that began in 2016. In 2021, the Wall Street Journal reported that Instagram, which had been acquired by Facebook in 2012, had conducted research showing that the app was toxic to the mental health of teenage girls.

Defenders of Facebook say that these impacts were unintended and unforeseeable. Critics claim that, instead of moving fast and breaking things, social media companies should have proactively avoided ethical catastrophe. But both sides agree that new technologies can give rise to ethical nightmares, and that should make business leaders — and society — very, very nervous.

We are at the beginning of another technological revolution, this time with generative AI — models that can produce text, images, and more. It took just two months for OpenAI’s ChatGPT to pass 100 million users. Within six months of its launch, Microsoft released ChatGPT-powered Bing; Google demoed its latest large language model (LLM), Bard; and Meta released LLaMA. ChatGPT-5 will likely be here before we know it. And unlike social media, which remains largely centralized, this technology is already in the hands of thousands of people. Researchers at Stanford recreated ChatGPT for about $600 and made their model, called Alpaca, open-source. By early April, more than 2,400 people had made their own versions of it.

While generative AI has our attention right now, other technologies coming down the pike promise to be just as disruptive. Quantum computing will make today’s data crunching look like kindergarteners counting on their fingers. Blockchain technologies are being developed well beyond the narrow application of cryptocurrency. Augmented and virtual reality, robotics, gene editing, and too many others to discuss in detail also have the potential to reshape the world for good or ill.

If precedent serves, the companies ushering these technologies into the world will take a “let’s just see how this goes” approach. History also suggests this will be bad for the unsuspecting test subjects: the general public. It’s hard not to worry that, alongside the benefits they’ll offer, the leaps in technology will come with a raft of societal-level harm that we’ll spend the next 20-plus years trying to undo.

It’s time for a new approach. Companies that develop these technologies need to ask: “How do we develop, apply, and monitor them in ways that avoid worst-case scenarios?” And companies that procure these technologies and, in some cases, customize them (as businesses are doing now with ChatGPT) face an equally daunting challenge: “How do we design and deploy them in a way that keeps people (and our brand) safe?”

In this article, I will try to convince you of three things: First, that businesses need to explicitly identify the risks posed by these new technologies as ethical risks or, better still, as potential ethical nightmares. Ethical nightmares aren’t subjective. Systemic violations of privacy, the spread of democracy-undermining misinformation, and serving inappropriate content to children are on everyone’s “that’s terrible” list. I don’t care which end of the political spectrum your company falls on — if you’re Patagonia or Hobby Lobby — these are our ethical nightmares.

Second, that by virtue of how these technologies work — what makes them tick — the likelihood of realizing ethical and reputational risks has massively increased.

Third, that business leaders are ultimately responsible for this work, not technologists, data scientists, engineers, coders, or mathematicians. Senior executives are the ones who determine what gets created, how it gets created, and how carefully or recklessly it is deployed and monitored.

These technologies introduce daunting possibilities, but the challenge of facing them isn’t that complicated: Leaders need to articulate their worst-case scenarios — their ethical nightmares — and explain how they will prevent them. The first step is to get comfortable talking about ethics.

Business Leaders Can’t Be Afraid to Say “Ethics”

After 20 years in academia, 10 of them spent researching, teaching, and publishing on ethics, I attended my first nonacademic conference in 2018. It was sponsored by a Fortune 50 financial services company, and the theme was “sustainability.” Having taught courses on environmental ethics, I thought it would be interesting to see how corporations think about their responsibilities vis-à-vis their environmental impacts. When I got there, I found presentations on educating women around the globe, lifting people out of poverty, and contributing to the mental and physical health of all. Few were talking about the environment.

It took me an embarrassingly long time to figure out that in the corporate and nonprofit worlds, “sustainability” doesn’t mean “practices that don’t destroy the environment for future generations.” Instead it means “practices in pursuit of ethical goals” and an assertion that those practices promote the bottom line. As for why businesses didn’t simply say “ethics,” I couldn’t understand.

This behavior — of replacing the word “ethics” with some other, less precise term — is widespread. There’s environmental, social, and governance (ESG) investing, which boils down to investing in companies that avoid ethical risks (emissions, diversity, political actions, and the like) on the theory that those practices protect profits. Some companies claim to be “values driven,” “mission driven,” or “purpose driven,” but these monikers rarely have anything to do with ethics. “Customer obsessed” and “innovation” aren’t ethical values; a purpose or mission can be completely amoral (putting immoral to the side). So-called “stakeholder capitalism” is capitalism tempered by a vague commitment to the welfare of unidentified stakeholders (as though stakeholder interests do not conflict). Finally, the world of AI ethics has grown tremendously over the last five years or so. Corporations heard the call, “We want AI ethics!” Their distorted response is, “Yes, we, too, are for responsible AI!”

Ethical challenges don’t disappear via semantic legerdemain. We need to name our problems accurately if we are to address them effectively. Does sustainability advise against using personal data for the purposes of targeted marketing? When does using a black box model violate ESG criteria? What happens if your mission of connecting people also happens to connect white nationalists?

Let’s focus on the move from “AI ethics” to “responsible AI” as a case study on the problematic impacts of shifting language. First, when business leaders talk about “responsible” and “trustworthy” AI, they focus on a broad set of issues that include cybersecurity, regulation, legal concerns, and technical or engineering risks. These are important, but the end result is that technologists, general counsels, risk officers, and cybersecurity engineers focus on areas they are already experts on, which is to say, everything except ethics.

Second, when it comes to ethics, leaders get stuck at very high-level and abstract principles or values — on concepts such as fairness and respect for autonomy. Since this is only a small part of the overall “responsible AI” picture, companies often fail to drill down into the very real, concrete ways these questions play out in their products. Ethical nightmares that outstrip outdated regulations and laws are left unidentified, and just as probable as they were before a “responsible AI” framework is deployed.

Third, the focus on identifying and pursuing “responsible AI” gives companies a vague goal with vague milestones. AI statements from organizations say things like, “We are for transparency, explainability, and equity.” But no company is transparent about everything with everyone (nor should it be); not every AI model needs to be explainable; and what counts as equitable is highly contentious. No wonder, then, that the companies that “commit” to these values quickly abandon them. There are no goals here. No milestones. No requirements. And there’s no articulation of what failure looks like.

But when AI ethics fail, the results are specific. Ethical nightmares are vivid: “We discriminated against tens of thousands of people.” “We tricked people into giving up all that money.” “We systematically engaged in violating people’s privacy.” In short, if you know what your ethical nightmares are then you know what ethical failure looks like.

Where Digital Nightmares Come From

Understanding how emerging technologies work — what makes them tick — will help explain why the likelihood of realizing ethical and reputational risks has massively increased. I’ll focus on three of the most important ones.

Artificial intelligence.

Let’s start with a technology that has taken over the headlines: artificial intelligence, or AI. The vast majority of AI out there is machine learning (ML).

“Machine learning” is, at its simplest, software that learns by example. And just as people learn to discriminate on the basis of race, gender, ethnicity, or other protected attributes by following examples around them, software does, too.

Say you want to train your photo recognition software to recognize pictures of your dog, Zeb. You give that software lots of examples and tell it, “That’s Zeb.” The software “learns” from those examples, and when you take a new picture of your dog, it recognizes it as a picture of Zeb and labels the photo “Zeb.” If it’s not a photo of Zeb, it will label the file “not Zeb.” The process is the same if you give your software examples of what “interview-worthy” résumés look like. It will learn from those examples and label new résumés as being “interview-worthy” or “not interview-worthy.” The same goes for applications to university, or for a mortgage, or for parole.

In each case, the software is recognizing and replicating patterns. The problem is that sometimes those patterns are ethically objectionable. For instance, if the examples of “interview-worthy” résumés reflect historical or contemporary biases against certain races, ethnicities, or genders, then the software will pick up on it. Amazon once built a résumé-screening AI. And to determine parole, the U.S. criminal justice system has used prediction algorithms that replicated historical biases against Black defendants.

It’s crucial to note that the discriminatory pattern can be identified and replicated independently of the intentions of the data scientists and engineers programming the software. In fact, data scientists at Amazon identified the problem with their AI mentioned above and tried to fix it, but they couldn’t. Amazon decided, rightly, to scrap the project. But had it been deployed, an unwitting hiring manager would have used a tool with ethically discriminatory operations, regardless of that person’s intentions or the organization’s stated values.

Discriminatory impacts are just one ethical nightmare to avoid with AI. There are also privacy concerns, the danger of AI models (especially large language models like ChatGPT) being used to manipulate people, the environmental cost of the massive computing power required, and countless other use-case-specific risks.

Quantum computing.

The details of quantum computers are exceedingly complicated, but for our purposes, we need to know only that they are computers that can process a tremendous amount of data. They can perform calculations in minutes or even seconds that would take today’s best supercomputers thousands of years. Companies like IBM and Google are pouring billions of dollars into this hardware revolution, and we’re poised to see increased quantum computer integration into classical computer operations every year.

Quantum computers throw gasoline on a problem we see in machine learning: the problem of unexplainable, or black box, AI. Essentially, in many cases, we don’t know why an AI tool makes the predictions that it does. When the photo software looks at all those pictures of Zeb, it’s analyzing those pictures at the pixel level. More specifically, it’s identifying all those pixels and the thousands of mathematical relations among those pixels that constitute “the Zeb pattern.” Those mathematical Zeb patterns are phenomenally complex — too complex for mere mortals to understand — which means that we don’t understand why it (correctly or incorrectly) labeled this new photo “Zeb.” And while we might not care about getting explanations in the case of Zeb, if the software says to deny someone an interview (or a mortgage, or insurance, or admittance) then we might care quite a bit.

Quantum computing makes black box models truly impenetrable. Right now, data scientists can offer explanations of an AI’s outputs that are simplified representations of what’s actually going on. But at some point, simplification becomes distortion. And because quantum computers can process trillions of data points, boiling that process down to an explanation we can understand — while retaining confidence that the explanation is more or less true — becomes vanishingly difficult.

That leads to a litany of ethical questions: Under what conditions can we trust the outputs of a (quantum) black box model? What are the appropriate benchmarks for performance? What do we do if the system appears to be broken or is acting very strangely? Do we acquiesce to the inscrutable outputs of the machine that has proven reliable previously? Or do we eschew those outputs in favor of our comparatively limited but intelligible human reasoning?

Blockchain.

Suppose you and I and a few thousand of our friends each have a magical notebook with the following features: When someone writes on a page, that writing simultaneously appears in everyone else’s notebook. Nothing written on a page can ever be erased. The information on the pages and the order of the pages is immutable; no one can remove or rearrange the pages. A private, passphrase-protected page lists your assets — money, art, land titles — and when you transfer an asset to someone, both your page and theirs are simultaneously and automatically updated.

At a very high level, this is how blockchain works. Each blockchain follows a specific set of rules that are written into its code, and changes to those rules are decided by whomever runs the blockchain. But just like any other kind of management, the quality of a blockchain’s governance depends on answering a string of important questions. For example: What data belongs on the blockchain, and what doesn’t? Who decides what goes on? What are the criteria for what is included? Who monitors? What’s the protocol if an error is found in the code of the blockchain? Who makes decisions about whether a structural change should be made to a blockchain? How are voting rights and power distributed?

Bad governance in blockchain can lead to nightmare scenarios, like people losing their savings, having information about themselves disclosed against their wills, or false information loaded onto people’s asset pages that enables deception and fraud.

Blockchain is most often associated with financial services, but every industry stands to integrate some kind of blockchain solution, each of which comes with particular pitfalls. For instance, we might use blockchain to store, access, and distribute information related to patient data, the inappropriate handling of which could lead to the ethical nightmare of wide-scale privacy violations. Things seem even more perilous when we recognize that there isn’t just one type of blockchain, and that there are different ways of governing a blockchain. And because the basic rules of a given blockchain are very hard to change, early decisions about what blockchain to develop and how to maintain it are extremely important.

These Are Business, Not (Only) Technology, Problems

Companies’ ability to adopt and use these technologies as they evolve will be essential to staying competitive. As such, leaders will have to ask and answer questions such as:

• What constitutes an unfair or unjust or discriminatory distribution of goods and services?
• Is using a black box model acceptable in this context?
• Is the chatbot engaging in ethically unacceptable manipulation of users?
• Is the governance of this blockchain fair, reasonable, and robust?
• Is this augmented reality content appropriate for the intended audience?
• Is this our organization’s responsibility or is it the user’s or the government’s?
• Does this place an undue burden on users?
• Is this inhumane?
• Might this erode confidence in democracy when used or abused at scale?

Why does this responsibility fall to business leaders as opposed to, say, the technologists who are tasked with deploying the new tools and systems? After all, most leaders aren’t fluent in the coding and the math behind software that learns by example, the quantum physics behind quantum computers, and the cryptography that underlies blockchain. Shouldn’t the experts be in charge of weighty decisions like these?

The thing is, these aren’t technical questions — they’re ethical, qualitative ones. They are exactly the kinds of problems that business leaders — guided by relevant subject matter experts — are charged with answering. Off-loading that responsibility to coders, engineers, and IT departments is unfair to the people in those roles and unwise for the organization. It’s understandable that leaders might find this task daunting, but there’s no question that they’re the ones responsible.

The Ethical Nightmare Challenge

I’ve tried to convince you of three claims. First, that leaders and organizations need to explicitly identify their ethical nightmares springing from new technologies. Second, a significant source of risk lies in how these technologies work. And third, that it’s the job of senior executives to guide their respective organizations on ethics.

These claims fund a conclusion: Organizations that leverage digital technologies need to address ethical nightmares before they hurt people and brands. I call this the “ethical nightmare challenge.” To overcome it, companies need to create an enterprise-wide digital ethical risk program. The first part of the program — what I call the content side — asks: What are the ethical nightmares we’re trying to avoid, and what are their potential sources? The second part of the program — what I call the structure side — answers the question: How do we systematically and comprehensively ensure those nightmares don’t become a reality?

Content.

Ethical nightmares can be articulated with varying levels of detail and customization. Your ethical nightmares are partly informed by the industry you’re in, the kind of organization you are, and the kinds of relationships you need to have with your clients, customers, and other stakeholders for things to go well. For instance, if you’re a health care provider that has clinicians using ChatGPT or another LLM to make diagnoses and treatment recommendations, then your ethical nightmare might include widespread false recommendations that your people lack the training to spot. Or if your chatbot is undertrained on information related to particular races and ethnicities, and neither the developers of the chatbot nor the clinicians know this, then your ethical nightmare would be systematically giving false diagnoses and bad treatments to those who have already been discriminated against. If you’re a financial services company that uses blockchain to transact on behalf of clients, then one ethical nightmare might be the absence of an ability to correct mistakes in the code — a function of ill-defined governance of the blockchain. That could mean, for instance, being unable to call back fraudulent transfers.

Notice that articulating nightmares means naming details and consequences. The more specific you can get — which is a function of your knowledge of the technologies, your industry, your understanding of the various contexts in which your technologies will be deployed, your moral imagination, and your ability to think through the ethical implications of business operations — the easier it will be to build the appropriate structure to control for these things.

Structure.

While the methods for identifying the nightmares hold across organizations, the strategies for creating appropriate controls vary depending on the size of the organization, existing governance structures, risk appetites, management culture, and more. Companies’ overtures into this realm can be classified as either formal or informal. In an ideal world, every organization would take the formal approach. However, factors like limited time and resources, the rate at which a company (truly or falsely) believes it will be impacted by digital technologies, and business necessities in an unpredictable market sometimes make it reasonable to choose the informal approach. In those cases, the informal approach should be seen as a first step, and better than nothing at all.

The formal approach is systematic and comprehensive, and it takes a good deal of time and resources to build. In short, it centers around the ability to create and execute on an enterprise-wide digital ethical risk strategy. Broadly speaking, it involves four steps.

Education and alignment. First, all senior leaders need to understand the technologies enough that they can agree on what constitutes the ethical nightmares of the organization. Knowledge and the alignment of leaders are prerequisites for building and implementing a robust digital ethical risk strategy.

This education can be achieved by executive briefings, workshops, and seminars. But it should not require — or try to teach — math or coding. This process is for non-technologists and technologists alike to wrap their heads around what risks their company may face. Moreover, it must be about the ethical nightmares of the organization, not sustainability or ESG criteria or “company values.”

Gap and feasibility analyses. Before building a strategy, leaders need to know what their organization looks like and what the probability is of their nightmares actually happening. As such, the second step consists of performing gap and feasibility analyses of where your organization is now; how far away it is from sufficiently safeguarding itself from an ethical nightmare unfolding; and what it will take in terms of people, processes, and technology to close those gaps.

To do this, leaders must identify where their digital technologies are and where they will likely be designed or procured within their organization. Because if you don’t know how the technologies work, how they’re used, or where they’re headed, you’ll have no hope of avoiding the nightmares.

Then a variety of questions present themselves:

• What policies are in place that address or fail to address your ethical nightmares?
• What processes are in place to identify ethical nightmares? Do they need to be augmented? Are new processes required?
• What level of awareness do employees have of these digital ethical risks? Are they capable of detecting signs of problems early? Does the culture make it safe for them to speak up about possible red flags?
• When an alarm is sounded, who responds, and on what grounds do they decide how to move forward?
• How do you operationalize and harmonize digital ethical risk assessment relative to existing enterprise-risk categories and operations?

The answers to questions like these will vary wildly across organizations. It’s one reason why digital ethical risk strategies are difficult to create and implement: They must be customized to integrate with existing governance structures, policies, processes, workflows, tools, and personnel. It’s easy to say “everyone needs a digital ethical risk board,” in the model of the institutional review boards that arose in medicine to mitigate the ethical risks around research on human subjects. But it’s not possible to continue with “and every one of them should look like this, act like this, and act with other groups in the business like this.” Here, good strategy does not come from a one-size-fits-all solution.

Strategy creation. The third step in the formal approach is building a corporate strategy in light of the gap and feasibility analyses. This includes, among other things, refining goals and objectives, deciding on an approach to metrics and KPIs (for measuring both compliance with the digital ethical risk program and its impact), designing a communications plan, and identifying key drivers of success for implementation.

Cross-functional involvement is needed. Leaders from technology, risk, compliance, general counsel, and cybersecurity should all be involved. Just as important, direction should come from the board and the CEO. Without their robust buy-in and encouragement, the program will get watered down.

Implementation. The fourth and final step is implementation of the strategy, which entails reconfiguring workflows, training, support, and ongoing monitoring, including quality assurance and quality improvement.

For example, new procedures should be customized by business domain or by roles to harmonize them with existing procedures and workflows. These procedures should clearly define the roles and responsibilities of different departments and individuals and establish clear processes for identifying, reporting, and addressing ethical issues. Additionally, novel workflows need to seek an optimal balance of human-computer interaction, which will depend on the kinds of tasks and the relative risks involved, and establish human oversight of automated flows.

The informal approach, by contrast, usually involves the following endeavors: providing education and alignment on ethical nightmares by leaders; entrusting executives in distinct units of the business (such as HR, marketing, product lines, or R&D) with identifying the processes needed to complete an ethical nightmare check; and creating or leveraging an existing (ethical) risk board to advise various personnel — either on individual projects or at a more institutional scale — when ethical risks are detected.

This approach does not require official changes of policies, harmonization and integration among departments, official changes in governance structure, or similar actions. While it can pack a powerful punch, it’s neither systematic nor comprehensive. Ethical risks may fall between the cracks and land on the front page.

. . .

In my work, I’ve found that the vast majority of organizations are operated and led by good people who do not intend to harm anyone. But I’ve also noticed that “ethics” is a word most companies are uncomfortable using. It is considered subjective or “squishy,” and outside the purview of business.

Both these views are incorrect. Invading people’s privacy, automating discrimination at scale, undermining democracy, putting children in harm’s way, and breaching people’s trust are decidedly clear-cut. They’re ethical nightmares that pretty much everyone can agree on.

Instead of understanding their roles in all of this, and trying to prevent it, many leaders remain focused on business as usual: Roles and responsibilities are fixed. Quarterly reports are due. Shareholders are watching. People have day jobs; they can’t be guardians of the moral galaxy as well. In many ways, it’s not evil but standard operating procedure that is the enemy of the good, or at least of the not bad. The tools of yesterday might have required malicious intent by those who wielded them to wreak havoc, but today’s tools require no such thing.

If you give people the opportunity, the breathing room, to do the right thing, they’ll do it happily. Creating that opportunity means not only permitting but encouraging or requiring people to talk the language of ethical nightmares. Make it a priority. Weave it into existing enterprise strategy. Ensure that everyone in the organization can tell you its nightmares and can rattle off five or six things the company does at the everyday operational level to make sure they don’t happen.

Leaders need to understand that developing a digital ethical risk strategy is well within their capabilities. Most employees and consumers want organizations to have a digital ethical risk strategy. Management should not shy away.