Tag: analysis

PURLs

PURLs are only as good as the maintenance work that has gone into updating the underlying URLs when they inevitably change. And in the lucky cases where the underlying URL haven’t changed, all the work that has gone into managing the infrastructure behind that URL namespace in order for that URL to stay the same. how many of the PURLs still work? This is complex enough for an actual research project and not just a quick blog. Over in the notebook I started by sampling all the target URLs (N=405637 n=662). In the process I noticed that it was oversampling some domains quite a bit like my.yoolib.net. So I tried again, but instead of sampling all the URLs I sampled the PURL namespaces (N=21894, n=644) and picked a random URL from each PURL namespace. This seemed to work better but still seemed to oversample, with hostnames list http://www.olemiss.edu showing up quite a bit. It looks like they might create a new PURL namespace for every finding aid they put up.

Of course, testing whether a URL still works is surprisingly tricky business: the response could be 200 OK but say Not Found, or it could be a totally different page (content drift)

Most important century

The “most important century” series of blog posts argues that the 21st century could be the most important century ever for humanity, via the development of advanced AI systems that could dramatically speed up scientific and technological advancement, getting us more quickly than most people imagine to a deeply unfamiliar future.

The long-run future is radically unfamiliar. Enough advances in technology could lead to a long-lasting, galaxy-wide civilization that could be a radical utopia, dystopia, or anything in between.

The long-run future could come much faster than we think, due to a possible AI-driven productivity explosion.

The relevant kind of AI looks like it will be developed this century – making this century the one that will initiate, and have the opportunity to shape, a future galaxy-wide civilization.

These claims seem too “wild” to take seriously. But there are a lot of reasons to think that we live in a wild time, and should be ready for anything.

We, the people living in this century, have the chance to have a huge impact on huge numbers of people to come – if we can make sense of the situation enough to find helpful actions. But right now we aren’t ready for this.

Segway Postmortem

now that there’s Segway equivalents for $100, time for a retrospective.

The Segway’s delays, cost, weight, and battery problems all derived primarily from 1 issue: how beautifully engineered, perhaps overengineered, the Segway was. The Segway was almost absurdly well-made, with custom components and redundancies built into every system to avoid breakdowns and accidents. Segways even had 2 identical motors, attached to 2 separate batteries, just in case something failed. The effect of all this redundancy and extra weight was to cause the batteries to drain quickly—especially considering how early in the development of rechargeable batteries 2002 was. Ideally, you’d be able to swap a drained battery for a fresh one—except, of course, that the Segway’s battery compartment was hermetically sealed to make it waterproof. The death of the 1 guy who still loved Segways enough to invest in Segway, killed by his Segway, basically seemed to put a cap on the dark comedy. It was too expensive, it looked doofy, it was cursed. End of story.

Distributed Innovation

contra “innovation only happens in person” beliefs:

it’s not actually that hard to collaborate productively at a distance in academia, at least once you’ve gotten to know someone.

innovation requires ever more collaboration among specialists as knowledge accumulates.

Over time, falling travel and communication costs have increasingly favored building those teams by turning to remote colleagues with the right specialization.

Post-Revolution

This is not some dark new age of cancel culture, however, it’s just a return to normality. Those who grew up in the late 20th century were living in a highly unusual time, one that could never be sustained, a sexual and cultural revolution that began in 1963 or 1968. But it has ended and, as all revolutionaries must do after storming the Bastille, they have built Bastilles of their own. The new order has brought in numerous methods used by the old order to exert control — not just censorship, but word taboo and rituals which everyone is forced to go along with, or at least not openly criticise. You might call it the new intolerance, or woke extremism, but all societies need the policing of social norms.

Link Rot

I was able to analyze ~2m externally facing links found in NYT articles since its inception in 1996. We found that 25% of deep links have rotted. If you go back to 1998, 72% of the links are dead. More than 50% of all NYT articles that contain deep links have at least 1 rotted link. The benefits of the internet and web’s flexibility—including permitting the building of walled app gardens on top of them that reject the idea of a URL entirely—now come at great risk and cost to the larger tectonic enterprise to, in Google’s early words, “organize the world’s information and make it universally accessible and useful.” What are we going to do about the crisis we’re in? A complementary approach to “save everything” through independent scraping is for whoever is creating a link to make sure that a copy is saved at the time the link is made. Authors of enduring documents—including scholarly papers, newspaper articles, and judicial opinions—can ask Perma to convert the links included within them into permanent ones archived at perma.cc; participating libraries treat snapshots of what’s found at those links as accessions to their collections, and undertake to preserve them indefinitely. A technical infrastructure through which authors and publishers can preserve the links they draw on is a necessary start. But the problem of digital malleability extends beyond the technical. The law should hesitate before allowing the scope of remedies for claimed infringements of rights—whether economic ones like copyright or more personal, dignitary ones like defamation—to expand naturally as the ease of changing what’s already been published increases.

Lockdown Effectiveness

1: Various policies lumped together as “lockdowns” probably significantly decreased R. Full-blown stay-at home orders were less important than targeted policies like school closures and banning large gatherings. Talking about which ones were “good” or “bad” is an oversimplification compared to the more useful questions of when countries should have started vs. stopped each to be on some kind of Pareto frontier of lives saved vs. cost.

2: If Sweden had a stronger lockdown more like those of other European countries, it probably could have reduced its death rate by 50-80%, saving 2500+ lives.

3: On a very naive comparison, US states with stricter lockdowns had ~20% lower death rates than states with weaker ones, and ~0.6% more GDP decline. There are high error bars on both those estimates.

4: Judging lockdowns by traditional measures of economic significance, moving from a US red-state level of lockdown to a US blue-state level of lockdown is in the range normally associated with interventions that are debatably cost-effective/utility-positive, with error bars including “obviously good” and “pretty bad”. It’s harder to estimate for Sweden, but plausibly for them to move to a more European-typical level of lockdown in the early phase of the pandemic would have very much cleared the bar and been unambiguously cost effective/utility-positive.

5: It’s harder to justify strict lockdowns in terms of the non-economic suffering produced. Even assumptions skewed to be maximally pro-strict-lockdown, eg where strict lockdowns would have prevented every single coronavirus case, suggest that it would have taken dozens of months of somewhat stricter lockdown to save one month of healthy life. This might still be justifiable if present strict lockdowns now prevented future strict lockdowns (mandated or voluntary), which might be true in Europe but doesn’t seem as true in the US.

6: Plausibly, really well-targeted lockdowns could have been better along every dimension than either strong-lockdown areas’ strong lockdowns or weak-lockdown areas’ weak lockdowns, and we should support the people trying to figure out how to do that.

7: All of this is very speculative and affected by a lot of factors, and the error bars are very wide.

The Circles of Friendship

The innermost layer of 1.5 is the most intimate; clearly that has to do with your romantic relationships. The next layer of 5 is your shoulders-to-cry-on friendships. They are the ones who will drop everything to support us when our world falls apart. The 15 layer includes the previous 5, and your core social partners. They are our main social companions, so they provide the context for having fun times. The next layer up, at 50, is your big-weekend-barbecue people. And the 150 layer is your weddings and funerals group who would come to your once-in-a-lifetime event. The layers come about primarily because the time we have for social interaction is not infinite. You have to decide how to invest that time, bearing in mind that the strength of relationships is directly correlated with how much time and effort we give them.

While I quibble with the names and sizes of these circles, it still seems directionally correct.

MTA could 2x off-peak capacity for free

In New York, the frequency of a bus or subway service is regularly adjusted every 3 months to fine-tune crowding. Where Berlin has a fixed clockface timetable in which most trains run every 5 minutes all day, New York prefers to make small changes to the frequency of each service throughout the day based on crowding. The New York approach looks more efficient on paper, but is in fact the opposite. It leads to irregular frequencies whenever trains share tracks with other trains, and weakens the system by leading to long waits. But another problem that I learned about recently is that it is unusually inconvenient for labor, and makes the timetabling of trains too difficult. New York should timetable its trains differently. Berlin offers a good paradigm, but is not the only one. As far as reasonably practical, frequency should be on a fixed clockface timetable all day. This cannot be exactly 5 minutes in New York, because it needs more capacity at rush hour, but it should aim to run a fixed peak timetable and match off-peak service to peak service. It’s a large increase in service. Frequency-ridership spirals work in your favor here. Increases in service require small increases in expenditure, even assuming variable costs rise proportionately – but they in fact do not, since regularizing frequency around a consistent number and reducing the peak-to-base ratio make it possible to extract far more hours out of each train driver, as in Berlin. Net of the increase in revenue coming from better service, such a system is unlikely to cost more in public expenditure.

This remains true even assuming no pay cuts for drivers in exchange for better work conditions. Pay cuts are unlikely anyway, but improving the work conditions for workers, especially junior workers, does make it easy to hire more people as necessary. The greater efficiency of workers under consistent timetabling without constant fidgeting doesn’t translate to lower pay, but to much more service, in effect taking those 550 annual hours and turning them into 900 through much higher off-peak frequency. It may well reduce public expenditure: more service and thus greater revenue from passengers on the same labor force.

they’d have to stop constantly messing with the schedule, which is pure insanity.

Dancing With Systems

We can’t control systems or figure them out. But we can dance with them!

I already knew that, in a way before I began to study systems. I had learned about dancing with great powers from whitewater kayaking, from gardening, from playing music, from skiing. All those endeavors require one to stay wide-awake, pay close attention, participate flat out, and respond to feedback. It had never occurred to me that those same requirements might apply to intellectual work, to management, to government, to getting along with people.