It’s difficult to muster a lot of sympathy for Amazon founder Jeff Bezos, reportedly the world’s richest individual, whose divorce this summer from his wife of 25 years cost him an estimated $38 billion. But recent attacks on Bezos and Amazon for selling clothing made in Bangladesh are ill-informed and unfair.

The controversy began in October when a Wall Street Journal investigation revealed that Amazon sells clothes from Bangladeshi factories that other retailers have blacklisted for poor safety records. Amazon’s critics ignore that the blacklisting these factories denies Bangladeshi garment workers the opportunity to earn a living.

Concerns about the safety of Bangladeshi garment factories stem from the 2013 Rana Plaza factory collapse, which killed more than 1,100 workers. In the aftermath of the collapse, the Accord on Fire and Building Safety in Bangladesh was established to promote factory safety and monitor compliance with safety agreements. The accord now includes more than 1,600 factories employing more than 2 million Bangladeshi workers.

Far more Bangladeshi firms, however, are not party to safety agreements under the accord. According to the International Labor Organization, there are more than 5,000 garment factories in Bangladesh; one study estimates there may be as many as 8,000 factories if subcontractors are counted. The 3,400 to 6,400 firms that aren’t part of the accord also employ about 2 million garment workers.

Last February, I participated in a panel discussion at the University of Denver alongside Joris Oldenziel, the accord’s deputy director for implementation. Understandably, perhaps, Oldenziel believes all Bangladeshi factories should have to live up to the accord’s safety standards. Similarly, the Wall Street Journal report implied that Amazon was harming workers by sourcing from factories blacklisted by the accord. Both are wrong.

With a per capita annual income of just over $1,500, poverty is widespread in Bangladesh. Factory work, even in unsafe factories, is a big step up for workers compared to subsistence agriculture, household services, or most other available alternatives. As I pointed out in my book, Out of Poverty: Sweatshops in the Global Economy, even the workers at garment factories that U.S. activists protest and label as harmful “sweatshops,” on average, earn far more than the “less than $2 a day” the vast majority of Bangladeshis were earning at the time of my analysis in 2010.

Better safety is not free, and companies respond to increased costs by shifting production to cheaper labor from other parts of the world or by investing in machinery and other automated processes that reduce employment. Forcing all companies to meet accord safety standards, either by passing new safety laws or by shaming companies like Amazon, jeopardizes jobs that provide millions of Bangladeshis the opportunity to escape extreme poverty.

When the overall pay is low, workers prefer the vast majority of their compensation to come in the form of wages, rather than safety expenditures or other benefits. That’s because they are in a struggle just to feed and clothe their families.

When productivity rises, so does total worker compensation. And as workers become wealthier, they demand more of their compensation in the form of safety and benefits. This process is underway in Bangladesh, and the accord is part of it.

Today, about only 10% of the employed population in Bangladesh is living on less than $1.90 per day. As productivity and compensation have risen, safety has also improved. The accord provides credible information on factory safety standards—or the lack thereof—giving workers more information to choose where they want to work.

The factories participating in the accord are disproportionately the larger firms with higher revenues. To force accord safety standards on smaller firms, where many of the poorer and less productive Bangladeshi workers are employed, would result in job losses for those who need jobs the most.

Sustainable safety improvements come through the market’s process of economic development. Amazon and other U.S. companies sourcing from accord factories, unlisted factories, and blacklisted factories are part of this process. If we care about the welfare of Bangladeshi workers, we need to continue to let the process play out.