Sheetal is a 42-year-old beauty gig worker who lives with her husband, two children, and parents-in-law in Dharavi, Mumbai. As a teenager, she learnt beauty skills from her sister (a make-up artist in the Bollywood film industry) and helped her co-run a salon. However, an arranged marriage put a pause on her ambitions, as her husband did not want her to work. Over the next couple of decades, marriage and childbearing created a persistent gap in Sheetal’s work life that she fought hard against through covert freelancing and upskilling. During the pandemic lockdown, facing down her physically and financially abusive husband, Sheetal decided enough was enough and joined Beauticare, a major beauty platform, through a referral from her friend. To pay for Beauticare’s compulsory training program and in-house beauty kit (which cost INR 52,000), she used money she had secretly saved through a local women’s self-help group. This decision was hard-won and required months of bargaining with her family, as well as extra planning to complete household chores by 12 P.M. every day so that she could go out to do gig work after. Things were going well at first. But around a year and a half into the job, Sheetal suddenly got a warning on her platform app that her average customer rating, at 4.6, had fallen below the new cut-off point of 4.75 out of 5. If she did not bring it up quickly, she would be fired. Two weeks later, Sheetal woke up to find that she was “ID blocked” from the app and all of her data. There was no redressal mechanism and nobody to file a complaint with.
Platform managers found that the average customer rating is highly correlated with the Net Promoter Score (NPS), which is a measure of how likely a customer is to continue using the platform’s services.
Sheetal was one of the first gig workers I met when I set out to study beauty work. She insisted that I shadow her and assist her to truly understand what the life of a gig worker is like. When she got laid off, we reached out to other gig workers to make sense of what had happened, and soon realized that this incident was not an isolated one. The abrupt change in the cut-off rating had impacted hundreds, if not thousands, of women gig workers across the country, causing mass layoffs. Platform offices were now flooded with gig workers who asked managers to explain why the cut-off rating had been changed and demanded another chance to bring up their rating. More than anything, they wanted to understand what had happened. But a clear explanation was out of reach. Managers were unwilling to explain the decisions behind the rating system, insinuating that it was a change that came from above them, facilitated by an algorithmic system. Since it was a machine-made decision, they could not be held accountable for it. “This is out of our hands, we cannot do anything, sorry,” we heard over and over.
In the following months, many theories and suspicions about the customer rating system spread among gig workers, including a widely held belief that platforms artificially deflate workers’ five-star ratings, making it impossible for them to raise the average rating beyond a point. As a researcher accompanying Sheetal and others to the platform office repeatedly to support their re-applications, it bothered me that there was no explanation or accountable figure in sight. I became preoccupied with tracing the chain of managerial decisions that had led to this outcome.
Investigating the customer ratings system
Platform companies use a host of metrics to track, control, and manage gig workers. The customer ratings system is a central metric through which platforms discipline workers into achieving shifting work targets, while also offloading the work of evaluating workers to the customers. Meanwhile, customers, realizing the power that the ratings system gives them, can pressure gig workers for free or extra services in exchange for a fair rating or discriminate against them by giving them unfair ratings. Since gig workers cannot opt out or contest the terms of the system if they want to stay employed, both platforms and customers use this system to exploit gig labor, with any penalties falling squarely on the workers.
In a country like India, where ~90% of all workers are in the informal economy, and women’s workforce participation is very low, platforms are celebrated for promising flexible, low-barrier work to groups that have historically struggled to secure decent employment. By examining the conditions around gig work more closely, through an investigation of the customer ratings system, we can see that many of these platform promises turn out to be false fronts for unchecked labor exploitation. After the mass layoffs, I began to speak to gig workers and platform managers about their views on the ratings system. As I investigated the chain of events behind the ratings cut-off, I made two discoveries that explained what had happened.
Platform managers found that the average customer rating is highly correlated with the Net Promoter Score (NPS), which is a measure of how likely a customer is to continue using the platform’s services.
First, the change in cut-off ratings was indeed algorithmically mediated. Platform managers found that the average customer rating is highly correlated with the Net Promoter Score (NPS), which is a measure of how likely a customer is to continue using the platform’s services. The NPS is considered a reliable predictor of customer retention, so in order to maintain their customer base, the platform sets the minimum average customer rating in relation to the Net Promoter Score through an algorithmic model. As platforms grow, they need to build and maintain a reliable customer base and adjust the NPS and rating cut-off point accordingly. This is precisely what happened here — the layoffs took place at the same time that the platform was shifting to a higher growth model. Except that gig workers were not consulted before the change was made, nobody explained this change to them, nor were they given a reasonable amount of time to bring up their ratings. Essentially, women gig workers were set up to fail and left to manage the fallout alone.
Except that gig workers were not consulted before the change was made, nobody explained this change to them, nor were they given a reasonable amount of time to bring up their ratings.
Second, I realized that gig workers were never provided with a basic understanding of how the customer ratings system worked (apart from a short explainer video on the app). This not only gave rise to speculation and confusion, but it also made it difficult for workers to respond strategically to changes in the cut-off point. Most gig workers did not know that the average customer rating is calculated as the simple average of the last 100 ratings received by a gig worker (it used to be an average of all the ratings ever received by a worker, but this was later changed to make the system fairer). Like a college CGPA, where a single D grade will drastically bring down the CGPA, and it takes many, many sustained As to increase it beyond a point; similarly, a few five-star ratings are not enough to bring up the average rating quickly. Gig workers needed to know this. Before the layoffs, the rating cut-off point was 4.5 out of 5. If a worker’s rating slipped below 4.5, they would be given a prompt warning, three chances to retrain with the company, and 20 gigs to pull up the rating, failing which they would be permanently blocked from work. When the cut-off point was abruptly changed from 4.5 to 4.75, workers were given two weeks to improve their ratings, not nearly enough time, even if they worked constantly. Most gig workers caught between the two cut-off points eventually ended up losing their jobs.
Opacity by design: Why we need to recenter the human employers at the heart of AI-mediated decisions
Sheetal’s experience of “ID blocking” highlights the role of AI-mediated platform management decisions in locking vulnerable workers out of work suddenly. It shows how platforms often rely on algorithmic variables to make decisions about gig workers; decisions that are hidden from gig workers until the last minute, which can force sudden exits from the workforce. This AI-mediated management decision robbed Sheetal and hundreds of other gig workers of their livelihoods, and it raises questions about explainability and accountability in the context of platform work, especially when the people most impacted get no visibility or say into the decision-making process.
This signals a dangerous new trend of employers using AI models as scapegoats to avoid accountability for deliberate human decisions that cause mass harm.
Following this incident, managers avoided gig workers’ complaints by shifting blame onto the algorithmic system behind the platform — a system that is opaque, inaccessible, and cannot speak for itself, but also a system that did not actually make the decision that led gig workers to be fired. This signals a dangerous new trend of employers using AI models as scapegoats to avoid accountability for deliberate human decisions that cause mass harm. Around the world, employers are now using this tactic to impose a range of harmful decisions that adversely affect workers, communities, and our environment, without offering any redressal or accountability mechanisms. In such a landscape, algorithmic systems are not just neutral tools to maximize efficiency but contested instruments of control that carry specific ideological preferences. They are being used strategically to shift the balance of power away from workers and concentrate it in the hands of employers.
This question is of increasing importance as AI models are being rapidly embedded into our economic, bureaucratic, and educational systems without any meaningful civic discussion or consent, and without regard to the long-term harms.
The AI-mediated layoffs at Beauticare were set in motion by a relatively short and clear chain of events taking place within a fairly simple system, and yet it was so easy for managers to hide the human decision behind immense amounts of opacity and unexplainability, and to avoid taking responsibility. What happens in more complex and layered AI-mediated systems, where AI models are used to help make a wide range of organizational decisions affecting how workers are hired, when they are fired, and everything in between? How much harm are we willing to risk before we hold the human beings behind these systems accountable? This question is of increasing importance as AI models are being rapidly embedded into our economic, bureaucratic, and educational systems without any meaningful civic discussion or consent, and without regard to the long-term harms. Just as I was writing this, Meta fired 10% of its tech workforce to offset the costs posed by rising AI infrastructures. Algorithmic management practices are now being used to exploit not only gig workers in the Majority World, but are being turned back on to the very classes of tech workers who built these surveillance systems in Silicon Valley and thought themselves immune to their effects. There may be poetic irony here, but more importantly, the moment calls for building solidarity between these two long-siloed classes of workers. To move towards equitable work futures for all, we must work together to re-center and visibilize the human decision-making that lies at the heart of the AI economy.
All names of individuals and platforms have been changed.
