Open sourcing the metafilter code base January 26, 2017 5:26 PM Subscribe
It’s been a little over ten years since the subject was last raised (at least based on a search for “open source”), but I’m curious if there’s been any thoughts about open-sourcing the MetaFilter codebase or admin tools. (I imagine the answer is “no”, especially to that last one.)
I don't have any plans to, no.
It is something I've thought about some and chattered about occasionally in MetaTalk discussions here and there, and there are arguments for going open-source that I like and I'm generally an advocate of doing so for projects where it's practical. So it's not an ideological no so much as a practical one.
Because my main motivation for open-sourcing the code would be to make it available as a tool for other folks—community members, motivated volunteer coders in the greater MeFi sphere, non-MeFi folks looking for a useful tool to work with or fork.
And the reality is: our codebase is a really idiosyncratic pile of interlocking pieces built very custom for our needs, on a not very popular platform, with going on 18 years now of iterative development across multiple sets of hands without anything like a real design plan or project management philosophy behind it. In other words, kind of a mess. It works for us, it does what the MetaFilter team and community needs it to do, and we continue to try to improve and sanitize it as time goes on, but it's not lean and it's not really fit for public consumption or a good fit for forking off into an unrelated project.
posted by cortex (staff) at 5:34 PM on January 26, 2017 [13 favorites]
It is something I've thought about some and chattered about occasionally in MetaTalk discussions here and there, and there are arguments for going open-source that I like and I'm generally an advocate of doing so for projects where it's practical. So it's not an ideological no so much as a practical one.
Because my main motivation for open-sourcing the code would be to make it available as a tool for other folks—community members, motivated volunteer coders in the greater MeFi sphere, non-MeFi folks looking for a useful tool to work with or fork.
And the reality is: our codebase is a really idiosyncratic pile of interlocking pieces built very custom for our needs, on a not very popular platform, with going on 18 years now of iterative development across multiple sets of hands without anything like a real design plan or project management philosophy behind it. In other words, kind of a mess. It works for us, it does what the MetaFilter team and community needs it to do, and we continue to try to improve and sanitize it as time goes on, but it's not lean and it's not really fit for public consumption or a good fit for forking off into an unrelated project.
posted by cortex (staff) at 5:34 PM on January 26, 2017 [13 favorites]
I've also thought about the idea of what a rebuild-from-scratch project with MeFi would look like, and gave it more serious thought for the first time in a long time when pb announced he was leaving, on the reasoning that if we felt like we really weren't able to find someone who could make the current system with its odd stew of tech requirements keep standing, buying time while we built out a new version that got 95% of the way there functionality-wise was a possibility.
That ended up not being necessary (thanks for being you, frimble!), but it was a serious consideration. And one of the things that I wrestled with on that front was whether and how building MeFi-as-product in that sense would be worth the effort: if we built something that was cleanly designed and engineered to do precisely what MeFi does, at MeFi's scale, would it be (a) general enough to work well for other people and (b) customizable enough to really replace the existing code base in a sane way? Or both? Or neither?
And that's what I come back to, and don't really have a firm answer for: is treating MetaFilter's code (either the current stuff or some notional from-scratch rebuild) as a generic CMS/BBS system rather than literally just as what works for MetaFilter a project that makes sense? Why that vs. some other extant CMS with a proper OSS team or even a commercial development team behind it? Who would use it, and why, and for what? How would the development work pay for itself? Etc.
A bunch of stuff that would need chewing on, even thinking about something more sensible than throwing the doors wide on the current codebase, which, again, I really can't see doing since we don't have any particular path forward to justify all the effort that would come with that.
posted by cortex (staff) at 5:42 PM on January 26, 2017 [6 favorites]
That ended up not being necessary (thanks for being you, frimble!), but it was a serious consideration. And one of the things that I wrestled with on that front was whether and how building MeFi-as-product in that sense would be worth the effort: if we built something that was cleanly designed and engineered to do precisely what MeFi does, at MeFi's scale, would it be (a) general enough to work well for other people and (b) customizable enough to really replace the existing code base in a sane way? Or both? Or neither?
And that's what I come back to, and don't really have a firm answer for: is treating MetaFilter's code (either the current stuff or some notional from-scratch rebuild) as a generic CMS/BBS system rather than literally just as what works for MetaFilter a project that makes sense? Why that vs. some other extant CMS with a proper OSS team or even a commercial development team behind it? Who would use it, and why, and for what? How would the development work pay for itself? Etc.
A bunch of stuff that would need chewing on, even thinking about something more sensible than throwing the doors wide on the current codebase, which, again, I really can't see doing since we don't have any particular path forward to justify all the effort that would come with that.
posted by cortex (staff) at 5:42 PM on January 26, 2017 [6 favorites]
As someone who strongly considered applying for the pb job, my thought is it would be cool to have access to the code in order to contribute, such as you could do with a private repo, but this opens a Pandora's Ratsnest of "what if someone wants to go beyond improvements to add a feature that has popular interest but is not in the plan by the powers that be," which is a pretty shitty debate for a community site to endure. I'm perfectly happy to take what I'm given, thereby.
posted by rhizome at 5:55 PM on January 26, 2017 [4 favorites]
posted by rhizome at 5:55 PM on January 26, 2017 [4 favorites]
It seems that the codebase is only part of what makes this site work as it does. The specific moment in time when it was created is a huge factor, but the human structures, relationships, rules, ways decisions are made, history that influences that, behavioral norms, ways those norms are enforced, and the financial structures are really what makes it what it is.
If there is ever an effort to re-code an infofilter code base, I hope there are some people willing to put in the time to document the non-software parts in a way that could act as a guide for others.
posted by amtho at 7:01 PM on January 26, 2017 [3 favorites]
If there is ever an effort to re-code an infofilter code base, I hope there are some people willing to put in the time to document the non-software parts in a way that could act as a guide for others.
posted by amtho at 7:01 PM on January 26, 2017 [3 favorites]
Just curious: Is the motivation behind this request to allow the unofficial creation of PoliticsMeFi or whatever?
posted by Harvey Kilobit at 7:39 PM on January 26, 2017 [2 favorites]
posted by Harvey Kilobit at 7:39 PM on January 26, 2017 [2 favorites]
That spurred my asking, yes. Because who wants to use vBulletin?
posted by Going To Maine at 8:27 PM on January 26, 2017 [1 favorite]
posted by Going To Maine at 8:27 PM on January 26, 2017 [1 favorite]
I keep wondering about creating a reproducible, local, moderated discovery-questions-discussion thing, personally.
posted by amtho at 9:10 PM on January 26, 2017
posted by amtho at 9:10 PM on January 26, 2017
The source for https://lobste.rs might be a reasonable alternative to vBulletin, etc. It's more like Hacker News than Metafilter, but it's not too far off and probably not hard to modify.
posted by Wobbuffet at 9:17 PM on January 26, 2017
posted by Wobbuffet at 9:17 PM on January 26, 2017
The thing is, for PoliFilter to be any more...responsive?..to posting frequency and topic proliferation than MeFi already, it's gonna have to be a webboard more or less anyway. VBulletin is worse than a (random distasteful image) deceased animal's testicles, but that's the general idea and I don't see much of an easy way around that. Discourse (bbs.boingboing.net) is along those lines as well. Could this be an ello thing?
posted by rhizome at 11:56 PM on January 26, 2017
posted by rhizome at 11:56 PM on January 26, 2017
Could this be an ello thing?
Is ello a thing?
posted by Going To Maine at 11:58 PM on January 26, 2017
Is ello a thing?
posted by Going To Maine at 11:58 PM on January 26, 2017
Having actually setup and run PoliticalFilter for a little while I'd say there are a number of things to consider:
1. How closely do you want your unofficial Metafilter related (UMR) site to resemble the main site?
2. Why should people visit your site, instead of Metafilter, for your UMR's particular niche? Will this put your UMR into competition with the main Metafilter site? If so, is that something you really want to do?
3. As to turning MeFi into web software and distributing it, is that the best use of site's limited sources?
4. Do you REALLY want to be responsible for maintaining a website 24/7 technically and keep it socially interesting enough to draw people to it?
5. What makes Metafilter unique is a combination of the site design (text only) and the moderators who run it, so how are you going to replicate that in your UMR?
posted by Brandon Blatcher at 6:57 AM on January 27, 2017
1. How closely do you want your unofficial Metafilter related (UMR) site to resemble the main site?
2. Why should people visit your site, instead of Metafilter, for your UMR's particular niche? Will this put your UMR into competition with the main Metafilter site? If so, is that something you really want to do?
3. As to turning MeFi into web software and distributing it, is that the best use of site's limited sources?
4. Do you REALLY want to be responsible for maintaining a website 24/7 technically and keep it socially interesting enough to draw people to it?
5. What makes Metafilter unique is a combination of the site design (text only) and the moderators who run it, so how are you going to replicate that in your UMR?
posted by Brandon Blatcher at 6:57 AM on January 27, 2017
As a little pet project I actually did this in late 2015— make a clone of MetaFilter using Python/Flask/SQLAlchemy/Postgres. It's now locked just for my reference due to creepiness and potential revenue pain for MetaFilter:
I used the infodump to build out the database schema using SQLAlchemy ORM, then loaded all the infodump data into it, then built /very/ similar layouts to Metafilter to show all the data.
Turns out, that's when you get into creepy territory. You tend to think of the infodump as something abstract, but when you give it a cloned interface, you can essentially become a user, rather then just report on them. So I could log in as `Going To Main` and I would be effectively you, seeing all your favourites, your posts, your history, in a way that feels quite different then the other infodump reporting apps people have created. So at that point I shot the admins a 'hey, is this cool?' question with a link to the temporary server.
The aim for me was to build the clone, then break it back down and use it as a little tutorial series/dev journal showing how similar the concepts are between all social sites, and it's just the little special-sauce that each one adds that makes them particularly unique (metafilters is the $5 and the moderation imo).
Cortex agreed it was a bit creepy, and asked if I were to proceed to use just my data from the infodump to populate, but I figured if you've got an importer build that brings in one user it can just as easily do all, so I thought better of it and just kept the repo private for my own reference.
The other, likely bigger reason that he brought up, and something to keep in mind if you're looking at the health of the community, is if I did make my code public again, or someone else builds an open clone, then some shitty person could grab it, load the infodump data in, wave a magic SEO stick based on some newly reversed engineered Google ranking trick and cause search-engine crawlers to have a issue where they downgrade the real MetaFilter's ranking and cause real financial harm.
posted by Static Vagabond at 7:49 AM on January 27, 2017 [7 favorites]
I used the infodump to build out the database schema using SQLAlchemy ORM, then loaded all the infodump data into it, then built /very/ similar layouts to Metafilter to show all the data.
Turns out, that's when you get into creepy territory. You tend to think of the infodump as something abstract, but when you give it a cloned interface, you can essentially become a user, rather then just report on them. So I could log in as `Going To Main` and I would be effectively you, seeing all your favourites, your posts, your history, in a way that feels quite different then the other infodump reporting apps people have created. So at that point I shot the admins a 'hey, is this cool?' question with a link to the temporary server.
The aim for me was to build the clone, then break it back down and use it as a little tutorial series/dev journal showing how similar the concepts are between all social sites, and it's just the little special-sauce that each one adds that makes them particularly unique (metafilters is the $5 and the moderation imo).
Cortex agreed it was a bit creepy, and asked if I were to proceed to use just my data from the infodump to populate, but I figured if you've got an importer build that brings in one user it can just as easily do all, so I thought better of it and just kept the repo private for my own reference.
The other, likely bigger reason that he brought up, and something to keep in mind if you're looking at the health of the community, is if I did make my code public again, or someone else builds an open clone, then some shitty person could grab it, load the infodump data in, wave a magic SEO stick based on some newly reversed engineered Google ranking trick and cause search-engine crawlers to have a issue where they downgrade the real MetaFilter's ranking and cause real financial harm.
posted by Static Vagabond at 7:49 AM on January 27, 2017 [7 favorites]
I don't think it would be terribly complicated to make a MetaFilter clone, particularly if you threw away any attempt at compatibility. Considering how (I guess) the biggest challenges of moderating political threads are deleting derails and subsequent followups rather than the occasional "WELL, ACTUALLY...", I'm not even sure why you'd want to copy MeFi. Even if the mods have some tools to streamline that process, users don't, and along browsers just giving up after the +1500 comments, you can have a lot of discussions going on at once, and reading those FPPs kind of turns into a masochistic pursuit.
However, you'd still need to find a way to pass site culture and moderation along with it. If you have the site culture and moderation, you could even use reddit codebase (not reddit.com, the code hosted somewhere else) and tweak the CSS to make it look more like MeFI to make things work. I'm sure that there's nothing inherently magic about the code that is what makes MeFi tick.
posted by lmfsilva at 9:22 AM on January 27, 2017
However, you'd still need to find a way to pass site culture and moderation along with it. If you have the site culture and moderation, you could even use reddit codebase (not reddit.com, the code hosted somewhere else) and tweak the CSS to make it look more like MeFI to make things work. I'm sure that there's nothing inherently magic about the code that is what makes MeFi tick.
posted by lmfsilva at 9:22 AM on January 27, 2017
I'm sure that there's nothing inherently magic about the code that is what makes MeFi tick.
Yep, basically this. Creating a new code base that does the basic stuff that the site does—that core 95% of posting, commenting, and basic admin tools that makes up most of what mods and users do here—wouldn't be difficult. Getting all the detailed bits documented and reimplemented to work the same way would be a lot more effort but that's more reflective of subjective/idiosyncratic design decisions over the years than any magical "this is clearly the best/only way to do this" insight. We've just built what works for us.
The closest thing to special sauce is probably the admin toolset and even that's not a matter of doing difficult things so much as just thinking to implement them at all. And it too is homegrown in a way that doesn't really translate to slam dunk "tear it out and distribute it" sort of treatment; we built tools around how we moderate, and how useful any given tool has been can vary over time as needs change and moderation processes evolve, etc.
There's also a lot of work that has gone into optimizing stuff like Recent Activity to work well for our particular db schema, our users' needs, and the specific scale of traffic and user activity the site operates at. It's tuned to MeFi pretty specifically and would be overbuilt for a lot of smaller things, and unworkable for a much larger level of activity.
posted by cortex (staff) at 9:50 AM on January 27, 2017 [2 favorites]
Yep, basically this. Creating a new code base that does the basic stuff that the site does—that core 95% of posting, commenting, and basic admin tools that makes up most of what mods and users do here—wouldn't be difficult. Getting all the detailed bits documented and reimplemented to work the same way would be a lot more effort but that's more reflective of subjective/idiosyncratic design decisions over the years than any magical "this is clearly the best/only way to do this" insight. We've just built what works for us.
The closest thing to special sauce is probably the admin toolset and even that's not a matter of doing difficult things so much as just thinking to implement them at all. And it too is homegrown in a way that doesn't really translate to slam dunk "tear it out and distribute it" sort of treatment; we built tools around how we moderate, and how useful any given tool has been can vary over time as needs change and moderation processes evolve, etc.
There's also a lot of work that has gone into optimizing stuff like Recent Activity to work well for our particular db schema, our users' needs, and the specific scale of traffic and user activity the site operates at. It's tuned to MeFi pretty specifically and would be overbuilt for a lot of smaller things, and unworkable for a much larger level of activity.
posted by cortex (staff) at 9:50 AM on January 27, 2017 [2 favorites]
this reminds me of the struggle Cmdr Taco had with opensourcing Slashdot's tangled web of perl CGI, which eventually became http://www.slashcode.com/.... and hasn't had a news update since 2009.
posted by jrishel at 11:59 AM on January 27, 2017 [5 favorites]
posted by jrishel at 11:59 AM on January 27, 2017 [5 favorites]
if I did make my code public again, or someone else builds an open clone, then some shitty person could grab it, load the infodump data in
Most metafilter pages are public, and trivial to iterate over (there's a reason modern sites tend to avoid exposing auto increment ids in their URLs), so not sure why that shitty person would have to wait for someone to release code...
posted by effbot at 6:06 AM on January 29, 2017
Most metafilter pages are public, and trivial to iterate over (there's a reason modern sites tend to avoid exposing auto increment ids in their URLs), so not sure why that shitty person would have to wait for someone to release code...
posted by effbot at 6:06 AM on January 29, 2017
I mean, we had that meta ages ago about a mysterious website that seemed to be mirroring all of metafilter's content. It's all feasible to do, the question is if it's worth it to make it easier. The answer seems to be "no".
posted by Going To Maine at 11:10 AM on January 29, 2017
posted by Going To Maine at 11:10 AM on January 29, 2017
Metafilter: we continue to try to improve and sanitize it as time goes on, but it's not lean and it's not really fit for public consumption
posted by tivalasvegas at 7:03 PM on January 29, 2017 [1 favorite]
posted by tivalasvegas at 7:03 PM on January 29, 2017 [1 favorite]
You are not logged in, either login or create an account to post comments
That said, something similar so someone could start a politics filter (or whatever) would probably go down well.
posted by freethefeet at 5:33 PM on January 26, 2017