Two Methods of Taking a look at a Blue Chicken: What’s Actually Occurring at Twitter?

“Twitter? Twitter? Twitter? Twitter?” “Um, Twitter’s sick. My greatest buddy’s sister’s boyfriend’s brother’s girlfriend heard from this man who is aware of this child who’s going with the lady who noticed Twitter go out at 31 Flavors final night time. I suppose it’s fairly severe.” That’s principally the place we’re with the protection. In the meantime: “‘Twitter Is Useless,’ 300 Million Individuals Submit On Twitter.”

I like to consider myself as technically literate, moreso again within the day than I’m now. Due to my Internet 1.0 heritage, I can arrange and run a web site, together with the backend. Nonetheless, the programs that Twitter engineers are (or have been) sustaining at scale are unimaginably complicated to me. These programs — their design and implementation, who runs them, what they ship to customers — are on the coronary heart of the Twitter story. Sadly, the primary two elements of Twitter are opaque simply now, so discovering out what’s actually occurring may be very troublesome (and on this story, Twitter the platform isn’t serving to very a lot). So what’s actually occurring?

Essentially the most primary query: Is Twitter down? The reply as of this writing (12:03PM EDT, November 21, 2022) is not any. Twitter isn’t down. Twitter’s APi web page says all programs are inexperienced:

Will Twitter go down? Down for Everybody or Simply Me? says it has, so likely it should:

Curiously, the final 5 outages are all since Musk’s ascension to CEO, however that’s way back to the info I can discover goes (and Twitter’s file of being up has been no means excellent. I can say, nonetheless, that as an especially on-line one that virtually lives on Twitter, that I didn’t really feel a factor).

On this publish, I’ll take a look at two elements of Twitter. First, I’ll take a look at Twitter’s software program, and present why the World Cup will stress it. Then, I’ll take a look at the gamers within the ongoing Twitter drama, and the function of every in Twitter’s present drama. I may also add an Appendix on Mastodon, Twitter’s open supply competitor.

Twitter Software program

The Twitter software program handles a lot of knowledge. From the Twitter Engineering Weblog, “Subsequent technology knowledge insights utilizing pure language queries“:

At Twitter, we course of roughly 400 billion occasions in actual time and generate petabyte (PB)-scale knowledge day by day.

That’s loads. Within the face of this knowledge stream, the important technical problem to protecting Twitter up — the baseline requirement for Twitter creating worth — is spikes. From the identical weblog, “When seconds actually do matter“:

On Twitter, the true world occurs in sub-minutely intervals and exercise on the platform follows. Whether or not it’s a retirement announcement from a member of a boy band, a authorities coup, a world broad soccer match purpose, or just the beginning of a brand new 12 months, site visitors patterns change rapidly and unpredictably as these occasions unfold. These are the moments when the minutely mannequin turns into inadequate. Let’s think about {that a} superstar couple publicizes the start of a kid at two seconds after the closest minute — for this instance, let’s say 2:00:02. With this announcement, a spike of site visitors causes undue strain on one of many many micro companies[1] that powers Twitter inflicting service errors to some customers.

The classical instance of this form of spike is the load on sewage programs when 1,000,000 bogs concurrently flush throughout a industrial break for the Superbowl. For Twitter, the 28 days of the World Cup pose an equal problem. From the New York Instances:

[The World Cup] is anticipated to deliver a deluge of site visitors to Twitter, which is the world’s fourth most visited web site, in response to Similarweb, a digital intelligence platform that tracks net site visitors. Twitter will get 6.9 billion visits every month, barely greater than Instagram’s 6.4 billion, although far fewer than Google, YouTube or Fb, in response to Similarweb estimates.

Previously, Twitter has a “Command Heart” to deal with the spikes, however many have left. From The Verge:

A number of members of Twitter’s “Command Heart” group, a bunch of engineers that’s on name 24/7 and acts because the clearing home for issues internally, additionally tweeted about their departures. “In the event that they go down, there is no such thing as a one to name when shit breaks,” stated an individual acquainted with how the group operates.

“An individual.” Many of the protection on inner Twitter points is single-sourced. One other single supply from the Guardian:

Twitter stands a 50% probability of a significant outage that might take the positioning offline through the World Cup, in response to a lately departed worker with data of how the corporate responds to large-scale occasions.

The previous worker, who was granted anonymity due to the sensitivity of what was mentioned, has data of the workings of Twitter Command Centre, the platform’s group of troubleshooters who monitor the positioning for points akin to site visitors spikes and knowledge centre outages.

Between the dearth of preparations and the dearth of staffing, I feel it’s going to be a tough World Cup for Twitter,” stated the previous worker.

He recommended that an incident of some variety – akin to a service responding slowly or incorrectly – is nearly a certainty through the 29-day [sic] competitors in Qatar, estimating a 90% risk of one thing going unsuitable that customers would see.

Properly, we’ll simply have to attend and see. 28 days shall be fairly a stress check.

From dealing with spikes, we flip to Twitter’s primary knowledge construction, the graph (or “social graph”). Right here’s a quite simple diagram of a really small graph:

Extra technically, as soon as once more from Twitter’s weblog, “Graph ML at Twitter.” The circles within the picture above are “nodes.” The strains between them are “edges”[2]:

Graphs are mathematical abstractions of complicated programs of relations and interactions. A graph represents a community of objects (known as nodes or vertices) with pairwise connections (edges). Graphs are ubiquitously utilized in fields as numerous as biology, quantum chemistry, and high-energy physics’. . Public conversations taking place on our platform usually generate a whole bunch of thousands and thousands of Tweets and Retweets day by day. This makes Twitter maybe one of many largest producers of graph-structured knowledge on the planet, second maybe solely to the Massive Hadron Collider.

“Second maybe solely to the Massive Hadron Collider.” Oof! Since Twitter as a enterprise exists to monetize relationships between customers as expressed by means of replies, Retweets, and Likes, the representational energy of the Twitter graph is of the primary significance. Little doubt that’s why Musk could enhance it. From Harvard Enterprise Faculty’s Andy Wu:

Doing that sort of change may be very, very harmful. So we’re going to have to attend and see what’s going to occur right here. My sense is that they’re prioritizing the interior core of the expertise of Twitter, the social graph.

So, what’s actually occurring with Twitter? Within the brief time period, an unlimited stress check of the software program: The World Cup. In the long run, bettering Twitter’s social graph.

Twitter Gamers

However what’s actually occurring? Let’s take a look at the gamers, and the struggles between them. The gamers fall into to buckets: The oligarchs, and the Skilled Managerial Class (PMC). Let’s take a look at every in flip.

First, the oligarchs. On this part, the set of all oligarchs has just one member: Musk. Software program luminary Grady Booch writes within the unrolled Twitter thread: “Twitter 2.0: a contemporary story about programs engineering“:

Take a legacy group that has through the years constructed a not excellent however fairly stable web-centric system at elastic world scale.

Let that engineering group function underneath the demonstrably, poor management of @jack and the distinct lack of grownup supervision by Twitter’s board for too a few years.

Herald a reluctant CEO with the charisma of Mussolini, the bluster of Trump, and the delicate nature of a bull in a china store.

Have that CEO finance the cope with a debt that might stun a group of ox, a debt that needs to be serviced on high of that group’s already unfavorable money stream.

On day one and for a lot of extra days that observe, have that CEO alienate his newly-acquired customers, scare away his few advertisers, and decimate the event group.

Have that CEO demand strict allegiance to his capricious edicts, institute unsustainable engineering processes, and tear aside the prevailing infrastructure by ripping out components he barely understands.

So right here’s the factor:

Making a Twitter 2.0 is just partly a software program engineering downside.

It’s a programs engineering downside that requires a agency and regular hand on all the opposite transferring components such because the enterprise mannequin, understanding the wants of its prospects and different stakeholders, defining a compelling imaginative and prescient for the long run.


Elon is approaching Twitter 2.0 as if it’s only a “hardcore software program engineering” downside.

That’s the place he’s gone off the rails.

Twitter 2.0 is actually not rocket science, however Elon is treating it like it’s. His give attention to the hardcore aspect of it’s comprehensible but misguided, and represents a harmful distraction and alternative value.

I don’t know how it will evolve within the coming weeks.

However I do know that it is going to be uncomfortable to look at, because the lives of Twitter’s employees are upended and as this essential world useful resource is trashed.

That is pretty much as good an outline of Musk’s scenario as I’ve seen, and from a Grasp of the Nice Runes, too. (In Musk’s protection, he’s clearly excited about the enterprise mannequin, too; introducing funds to creators — Twitter has many, many artists, writers, performers, and so forth. — could be a significant change to the enterprise mannequin, and a welcome one, too.)

Now, the PMC. For many who got here in late, the definition from Barbara and John Ehrenreich (1977):

Salaried psychological staff who don’t personal the technique of manufacturing and whose main perform within the social division of labor could also be described broadly because the copy of capitalist tradition and capitalist class relations.

Definitely programmers; definitely managers. First, the managers. Matt Stoller writes:

Whereas I’m no fan of Musk, I do know that Twitter was horribly managed earlier than his takeover.

Grady Booch (above) famous the “demonstrably, poor management of @jack and the distinct lack of grownup supervision by Twitter’s board.” And from HBS’s Andy Wu:

Twitter was truly in very unhealthy form and didn’t fairly have a future anyway. By way of actually powerful issues, [Musk] is the sort of CEO you in all probability must check out. Twitter is definitely a really, very troublesome enterprise problem that no one else has been in a position to resolve. So at this level, we would must, like, swing the automotive round and see what occurs.

This can be a firm that couldn’t ship an Edit button to customers for the complete interval of its existence, despite the fact that customers have been clamoring for it. (Mastodon, Twitter’s open supply competitor, has an edit button.)

Now the professionals. These may be divided into employees (inside Twitter) and the press (outdoors, reporting on it). So far as the employees goes, others echo Gooch’s grudging reward (“not excellent however fairly stable). Andy Wu as soon as extra:

The individuals who constructed the Twitter product positively constructed a fairly resilient infrastructure to outlive by means of all of this. The technical half can function with out plenty of the individuals.

An instance of such resilience: swish degradation (like when an internet web page masses a barely formatted model of itself, however nonetheless offers you one thing to learn:

The employees did some good engineering, there. (Different examples of the form of unsung enginering it takes to maintain a really massive system up and working right here, right here, and right here.) After all, from the oligarch’s perspective, none of this interprets straight into headcount[3],[4]. Musk could determine he can rent again what he wants:

Right here, for instance, is a really completely satisfied camper who stayed:

Let’s simply hope the Peter Precept doesn’t apply right here.

And now the second set of pros, the press. The important thing level right here is that the press have performed what old-school journalists ought to by no means do: They’ve turn out to be a part of the story by doing little or no reporting, in “print” or on Twitter, as a substitute participating in an unlimited dogpile of Elon Musk. (I discover the press’s pearl-clutching over a set of software program professionals transferring on to inexperienced fields and pastures new, and their silent indifference to, say, railroad employee or nurses, actually galling. I additionally discover the implication that Musk, and I suppose Trump, are the one billionaires worthy of censure galling as effectively.) For now, I’ll merely be aware the financial incentives. From the final sentence of this Instances story:

Mr. Musk has more and more downplayed the function of conventional media over the previous few months, citing Twitter as probably the greatest platforms for the rise in “citizen journalism,” as he put it.

Oh. Can’t have that. Arthur wouldn’t prefer it.


As normal, Atrios is smart. “The Chicken App Will Be Gone And We Will All Be Free“:

Twitter had two essential contributions to the world (effectively, the US, anyway): It killed the affect of Drudge….

Fairly proper. I bear in mind when Drudge drove the information cycle. Having Twitter drive the information cycle, for all its many faults, is a gigantic enchancment.

… and all of a sudden a bunch of immensely self-important individuals have been confronted with a whole bunch of individuals, typically deservedly, calling them shitheads day by day.

By calling them shitheads, I imply typically calling them shitheads backed up with good arguments. And plenty of these arguments have been from individuals of colour and different “riff raff” akin to individuals who went to lesser faculties than Harvard.

Trivially, Twitter permits me to name Rochelle Walensky a eugenicist to her face (or her intern’s face), and have others see it. To me, that’s what’s actually taking place at Twitter. I’m very doubtful that there’s one other platform that allows that. Non-trivially, Twitter allowed aerosol scientists — a lot of whom are from [shudder] state faculties — to name out the droplet dogmatists efficiently (“backed up with good arguments”). That’s an amazing achievement in human phrases, in medical phrases, in scientific phrases, in each means. We are going to see whether or not that’s the Twitter Musk preserves (as I do know the censorship-mad PMC won’t). We may also see how Twitter holds up underneath the World Cup stress check. Attention-grabbing instances!


[1] These are the microservices Musk calls “bloatware.” Individually, maybe. Within the combination, I’m not so positive. For instance, if Twitter’s copyright strike system was a microservice, there’s a difficulty:

Social media customers on Twitter can now add total motion pictures in threads as the corporate’s automated copyright enforcement/ takedown system now not appears to be purposeful, Forbes reported on Sunday.

A consumer uploaded the 2006 film, The Quick and the Livid: Tokyo Drift in two-minute video clippings unfold out in a thread of 49 tweets, which when viral on the platform. It wasn’t a one-off occasion that someway simply acquired by means of. One other consumer uploaded the film Hackers in a equally lengthy thread on Saturday, which even acquired 14,000 likes, Enterprise Insider reported.

Hollywood received’t like that. Nor will FIFA. However I’ve a suggestion:

I wager the pirated motion pictures wouldn’t go viral if the algo didn’t push them. Musk ought to flip off the algo! Give individuals what they need: A timeline of tweets from accounts that they observe, in reverse chronological order. This might take away an unlimited quantity of bloat.

[2] What I name a “yarn diagram” in conspiracy idea is a graph. The nodes are conspirators; the yarn is relationships. Once I say that the yarn in a yarn diagram is “too tight,” I imply that the creator of the graph hasn’t been nuanced sufficient about relationships (for instance, assuming orders are given when shared curiosity is sufficient).

[3] Or perhaps it does. From former staffer Dan Luu:

Another excuse to have in-house experience in numerous areas is that they simply pay for themselves, which is a particular case of the generic argument that giant firms ought to be bigger than most individuals count on as a result of tiny share good points are price a big quantity in absolute {dollars}. If, within the lifetime of the specialist group just like the kernel group, a single individual discovered one thing that persistently diminished TCO [Total Cost of Ownership] by 0.5%, that might pay for the group in perpetuity, and Twitter’s kernel group has discovered many such adjustments. Along with kernel patches that generally have that sort of influence, individuals may also discover configuration points, and so forth., which have that sort of influence.

[4] If Musk is true about headcount, and also you play the ponies:


Quite a lot of Twitter customers have ventured out into Mastodon, Twitter’s open-source competitor. Naturally, they reported their findings on Twitter.

Mastodon is massive by its personal requirements, however not by Twitter’s:

There’s one Twitter; there are lots of Mastodon servers (therefore jokes about garages). Meaning onboarding is tough. It’s a must to decide a server, then apply for an account, then wait. Some servers have shut down functions as a result of they’ll’t deal with the load;

The Mastodon servers are collectively often known as the “fediverse” (federated universe):

The fediverse isn’t easy:

Typically members of the fediverse battle:

After which there are the admins:

I’ve arrange an account on Mastodon. Whether or not I transfer my tiny presence there’s an open query…

Print Friendly, PDF & Email