Another HN tool moderators use that isn't mentioned in the post is the second-chance pool, where moderators will rescue a post which did not receive many upvotes and give it placement on the front page for a bit of time.
Where is this documented? Not trying to be snippy - I have no idea what powers the moderators have in this realm. Maybe HN could benefit from some lobste.rs-esque moderation transparency.
Even though HN is a community, the front page is essentially the mods' personal judgment. They are the editors, and HN is a newspaper.
I like it this way. They have excellent taste. The community has repeatedly proven it cannot be trusted to upvote intellectually gratifying content. pg's original HN code contained a flag for "rally", meaning the story is a rallying cry, and is therefore penalized because it's so easy to mindlessly upvote it.
The results speak for themselves: HN is now almost a top-1000 site by traffic.
They had to be phenomenal. I don't think many people appreciate just how difficult this was to pull off, or how much they work their asses off every single day. If they ignored HN for even a week, it would deteriorate noticeably.
Yet technology is at least as important. The fact that our upvotes do matter makes the situation unique -- although the mods can determine the relative weights of stories, we can still push a story upwards if enough of us are interested, and the story fits the site's original goal of gratifying intellectual curiosity.
There is no company anywhere in the world like this. At this point, HN practically defines our culture. When news stories say things like "so-and-so story about has been making the rounds on social media," at least some of them are talking about HN now. And your coworkers probably hear about the tech they use at least partly from HN.
The sole reason they're in this position is because they deserve to be. I hope they stay selective. As to being secretive, we should let them choose. If they want to reveal, fine. If not, fine. They've earned it. Even still, they've been remarkably transparent.
HN's cool but it's just one forum among a million others. The moderation is good though sometimes arbitrary. That's a lot easier for users to accept when they know why they are getting positively/negatively moderated, which is a good reason for more openness. But at the end of the day it's a forum, the moderators have the final say, that's the contract you agree to in using it.
If you don't like the way this site is edited, stand up an m1.large, hook Rails up to an sqlite database, and build your own. You'll be two generations of technology ahead of HN.
The freedom to take other people's private belongings and repurpose them to your own ends is no real kind of freedom at all. But that's ultimately what you're yearning for when you allude to the Empire of Napoleon.
Antirez tried, with lamernews: https://github.com/antirez/lamernews/blob/master/app.rb
Diff that vs HN circa 2008: https://github.com/evanrmurphy/SweetScript/blob/master/arc3....
Lamernews seems like just another HN clone, but back when it was launched it was a serious attempt at being a competitor. They probably came closest to pulling away a significant chunk of HN. I started to use it.
Then I ran into missing features. Little quality of life stuff. I forget what they were, but the whole site felt noticeably harder to use. So I stopped using it.
It's so tempting to think "HN is so simple. Just throw Rails at the problem and you can wipe the floor with them." But Lamernews was 2,964 lines of Ruby, and only managed to contain about 60% of the features. news.arc is 2,616 lines. And I don't care what anybody says: if you sat down and forced yourself to read that code for a week straight, you'd find it is actually quite readable. It's terse because it's for getting shit done.
I'm not saying you'll be as successful as HN. Lobste.rs, which is run carefully by thoughtful people using software which is probably better than HN's, is much less successful. But that's the point.
You said you'd be two generations of tech ahead of HN. I know it was tongue in cheek, and I'm not calling you out. I'm nerding out about how much functionality pg managed to pack into so few lines of code.
And it all worked! No data loss, no massive security problem, and the last time HN was down for >24h was because pg manually edited the database and accidentally created a circular comment chain.
One way to look at the situation is that pg's original vision for arc failed. Arc was supposed to be what Clojure turned out to be. But even though HN alone is the one significant Arc project, what a project! The world was made materially better thanks to it. And I don't think they could've pulled off HN in some other language. There was too much work for one person to do to be distracted by the thousands of details a modern webdev needs to stay on top of.
I think it's possible to do the same thing for modern webdev. Arc's goal was to be a reliable foundation for building webapps. It comes with a user+admin system out of the box, for example. Every Arc app does.
I've been playing around with hooking up AngularJS to it, because unlike React or Angular, AngularJS can operate purely in HTML. You normally never want to do that -- usually you want to set up controllers and such. But when you control the language, you end up being able to give it the Babel treatment: the output looks complex, but you get all the features of a modern frontend site. Initial experiments were quite promising.
And from there, it's easy to just emit JSX instead. React? Angular? Sure, whatever. It's treated as compiler output.
(Indirect sourcemaps are funny... JSX compiles to minified JS, whose sourcemap points back to the JSX. But if Arc spits out JSX, you'll want a sourcemap from JSX -> Arc as well. It's sourcemaps all the way down.)
You can call me out any time you'd like. I think an actual database backend for this site is only a couple years old. Or do we actually have one yet? I know the API has one, but HN doesn't run the API. It's possible this thing is still relying on flat files.
Flat files FTW. Proof that a simple solution can scale. And not just in a "Why the fuck would you subject yourself to this torture?" kind of way, but in a "This actually simplifies everything" way. It's why Arc was so fucking fast: remember how impactful memcached was? Arc had that from day one.
Here's how it works. Everything is loaded into memory at startup. You can think of it like a bigass JSON tree. Anytime you change the tree, write the whole thing out to disk. Since all the data is loaded from file "foo", and you write out to "foo.tmp" followed by `mv foo.tmp foo`, there is no risk of data loss, ever.
So what about sharding? Well, yeah, that's a problem. But it's not a problem that HN had to deal with. You can see how effective it was to throw a single 64GB RAM monster at the problem rather than a cluster of servers. The moment they hooked up Cloudflare to HN, they got all the same benefits a big cluster would provide anyway.
People can be prejudiced against this simplicity all they want, but it's hard to argue with results. You don't even need to know SQL to start using it.
"But migrations --" well, Arc makes it super easy to define a "template", which is just a set of fields, and each field is whatever you want them to be.
https://github.com/evanrmurphy/SweetScript/blob/6b4ed26acc9c...
(deftem item
id nil
type nil
by nil
ip nil
time (seconds)
url nil
title nil
text nil
votes nil ; elts each (time ip user type score)
score 0
sockvotes 0
flags nil
dead nil
deleted nil
parts nil
parent nil
kids nil
keys nil)
It's sort of similar to mongoose record definitions.
But look, HN is proof positive that this system can work and can scale to a top-1000 website. 99% of webapps won't even come close to those performance requirements. And when you do approach them, you can find ways to make it scale.
My main focus has been updating Arc to run on JS, and to add keyword arguments. That way you'll be able to use the whole JS ecosystem too.
For what it's worth, the published Arc/HN code is rapidly approaching a full decade out of date.
When you get the design right, there's no need to change it. :)
Jokes aside, I refuse to let Arc die. There were genuinely good ideas embedded in it, amidst the rough edges. People just never forced themselves to use it. It's like if Vim was invented but had no documentation, so no one ever realized it's powerful if you embrace it.
But yeah, I imagine my version will have a different name and a shiny website. Being a decade out of date does matter, especially for hiring purposes.
I think I'm just saying that I'm not sure you can conclude they don't have a real database backend now from the news.arc code.
Ah, yeah. (It's odd that Arc went closed-source but kept improving. It's maybe one of the most important custom languages in the industry, relative to HN's impact on the world.)
As far as I can tell, the arc forums site is still up and running, and a lot of people seem to be focused on Anarki[0].