4 month ago
nelson : WoW datacenter size - Works out to 6 CPUs and 10 gigabytes of RAM per 1000 subscribers
# copy6 month ago
bmilleare : CloudCrowd - If Carlsberg made worker/job queue servers...
Simon Willison : cloud-crowd - cloud-crowd. New parallel processing worker/job queue system with a strikingly elegant architecture. The central server is an HTTP server that manages job requests, which are farmed out to a number of node HTTP servers which fork off worker processes to
# copy13 month ago
Jeremy Zawodny : How FriendFeed uses MySQL to store schema-less data - How FriendFeed uses MySQL to store schema-less data: interesting approach that uses MySQL as more of a glorified column store, since writing "normal" relational queries becomes difficult. But it does scale and that's what matters.
Simon Willison : How FriendFeed uses MySQL to store schema-less data - How FriendFeed uses MySQL to store schema-less data. The pain of altering/ adding indexes to tables with 250 million rows was killing their ability to try out new features, so they’ve moved to storing pickled Python objects and manually creating the i
nelson : FriendFeed datastore - Using MySQL just to store python dicts
joshua : How FriendFeed uses MySQL to store schema-less data - Bret Taylor's blog - this is very much like the datastore i wanted to build for delicious. instead they built the usual crap.
jcgregorio : How FriendFeed uses MySQL to store schema-less data - Bret Taylor's blog
# copy13 month ago
Simon Willison : Building and Scaling a Startup on Rails: 12 Things We Learned the Hard Way - Building and Scaling a Startup on Rails: 12 Things We Learned the Hard Way. Lessons learned from Posterous. Some good advice in here, in particular “Memcache later: If you memcache first, you will never feel the pain and never learn how bad your data
philgyford : Building and Scaling a Startup on Rails: 12 Things We Learned the Hard Way - Axon Flux - A Ruby on Rails Blog - Nice "lessons learned" piece (via Simon Willison)
# copy14 month ago
Simon Willison : Infrastructure for Modern Web Sites - Infrastructure for Modern Web Sites. Leonard’s thoughts on what the next generation of web frameworks should aim to provide.
jcgregorio : Infrastructure for Modern Web Sites « random($foo) - Things you'll need.
Jeremy Zawodny : Infrastructure for Modern Web Sites - Infrastructure for Modern Web Sites: this is so dead on, it's not even funny
deusx : Infrastructure for Modern Web Sites « random($foo) - "One of the things that I did when I wrapping up at Yahoo! was to begin to take a look at the current state of web frameworks. I ended up picking Django, but I have to say, I was disappointed with the state of what’s out there. Friends will have he
# copy14 month ago
deusx : Joshua Schachter - Lessons Learned in Scaling and Building Social Systems | Kris Jordan - "Joshua Schachter is the creator of del.icio.us, creator of geoURL and co-creator of Memepool. Built delicious in 2003, sold it to Yahoo! in 2005, and left Yahoo! just a few years ago. 4+ million users. 100s of millions of urls indexed. He was there
# copy15 month ago
joshua : Spock Proxy - Spock Proxy supports range-based horizontal paritioning of a large MySQL database. The proxy intercepts SQL queries from the client, sends queries to the correct databases based on how the database is partitioned, then aggregates the results from each dat
deusx : Spock Proxy - "Spock Proxy supports range-based horizontal paritioning of a large MySQL database. The proxy intercepts SQL queries from the client, sends queries to the correct databases based on how the database is partitioned, then aggregates the results from ea
Simon Willison : Spock Proxy - Spock Proxy. A MySQL Proxy fork (no Lua) that concentrates solely on sharding, by parsing incoming SQL statements and redirecting them across multiple databases. There are some limitations on the SQL that can be handled (no nested queries, joins across
Rod Begbie : Spock Proxy - Proxy for transparent(-ish) sharding of MySQL databases. [via] #
# copy16 month ago
deusx : 7 Stages of Scaling Web Applications - SlideShare - "Slides from LinuxWorld presentation by John Engates, CTO of Rackspace."
# copy16 month ago
nelson : Twitter stats - 2-3 times as much traffic as normal and the site performed very well. Good job, guys.
# copy
18 month ago
deusx : Code: Flickr Developer Blog » Flickr Engineers Do It Offline - "It seems that using queuing systems in web apps is the new hottness . While the basic idea itself certainly isn’t new, its application to modern, large, scalable sites seems to be. At the very least, it’s something that deserves talking about
Simon Willison : Flickr Engineers Do It Offline - Flickr Engineers Do It Offline. Flickr wrote their own queuing mechanism (in PHP), and currently run ten queue servers on dedicated hardware for tasks like pushing new photos in to indexes, denormalisation and “backfills” which move data between clu
# copy
18 month ago
Anil : Details of Execution - Sometimes if you do something very difficult, and you do it really well, the end result is that your achievement becomes completely invisible. I mentioned a year and a half ago that I like Twitter. That was a little bit less common a position to take b
# copy
19 month ago
Anil : Details of Execution - Sometimes if you do something very difficult, and you do it really well, the end result is that your achievement becomes completely invisible. I mentioned a year and a half ago that I like Twitter. That was a little bit less common a position to take b
# copy
19 month ago
Anil : Details of Execution - Sometimes if you do something very difficult, and you do it really well, the end result is that your achievement becomes completely invisible. I mentioned a year and a half ago that I like Twitter. That was a little bit less common a position to take b
# copy19 month ago
Anil : Details of Execution - Sometimes if you do something very difficult, and you do it really well, the end result is that your achievement becomes completely invisible. I mentioned a year and a half ago that I like Twitter. That was a little bit less common a position to take b
# copy
21 month ago
Simon Willison : Queue everything and delight everyone - Queue everything and delight everyone. Les Orchard explains why I’ve been getting interested in queues recently: “One of the problems it seems most modern web apps face is the tendency to want to do everything all at once, and all in the same code t
Rod Begbie : Queue everything and delight everyone - Good thoughts from Les Orchard -- Your webapps don't have to do all your work the second a user clicks the button on a form. Queue up tasks and respond quickly, and everyone wins. #
Linkorama : Queue everything and delight everyone - The idea here is that the social structure can help you scale, while still delighting people.
# copy
22 month ago
bmilleare : sparrow - Sparrow is a really fast lightweight queue written in Ruby that speaks memcache.
# copy
22 month ago
joshua : Preparing For EC2 Persistent Storage
bmilleare : Preparing For EC2 Persistent Storage - Awesome write-up for preperation of persistent storage on AWS
# copy
23 month ago
deusx : I'm Going To Scale My Foot Up Your Ass - Ted Dziuba - "Shut up about scalability, no one is using your app anyway."
Jeremy Zawodny : I'm Going To Scale My Foot Up Your Ass - Ted Dziuba - I'm Going To Scale My Foot Up Your Ass - Ted Dziuba: hahahhahahahaahahhahha... so true
# copy
23 month ago
deusx : random($foo): Internet Asshattery, Armchair Scaling Experts Edition - "it really is true that if you haven't done it, that is: been intimately involved growing a social web app from prototype to large/giant-scale on a UNIX stack, then you really don't know shit."
Kellan : random($foo): Internet Asshattery, Armchair Scaling Experts Edition - I miss Leonard, Gordon, and Andy. But now that they’ve dispersed from Big Purple it’s nice to see all three of them schooling the internet in what real tech reporting might look like. You almost forget how bad it’s gotten until you see #
Simon Willison : Internet Asshattery, Armchair Scaling Experts Edition - Internet Asshattery, Armchair Scaling Experts Edition. Leonard says what needs to be said about the most recent case of Twitter scaling flame-bait. [via]
# copy
23 month ago
Jeremy Zawodny : Scaling out MySQL: Hardware today and tomorrow - Scaling out MySQL: Hardware today and tomorrow: good talk slides
bmilleare : Scaling out MySQL: Hardware today and tomorrow - Good slides.
# copy
23 month ago
Simon Willison : Amazon takes EC2 to the next level with persistent storage volumes - Amazon takes EC2 to the next level with persistent storage volumes. You can store a snapshot of a storage volume to S3 with a single API call, making backups trivial.
Jeremy Zawodny : Amazon takes EC2 to the next level with persistent storage volumes - Amazon takes EC2 to the next level with persistent storage volumes: good to hear how well baked this appears to be
bmilleare : Amazon takes EC2 to the next level with persistent storage volumes - This is potentially game-changing in the cloud computing dept.
# copy
24 month ago
deusx : Tom White: "Disks have become tapes" - "In essence MapReduce works by repeatedly sorting and merging data that is streamed to and from disk at the transfer rate of the disk. Contrast this to accessing data from a relational database that operates at the seek rate of the disk"
Matthew M. Boedicker : thinking of disks as a sequential device rather than a random access device - (via reddit) [via]
# copy
24 month ago
deusx : Working Notes on Consistent Hashing - Laughing Meme - "Nice to see consistent hashing go from obscure to blindingly obvious in a few short whitepapers. "
# copy
24 month ago
Simon Willison : Consistent Hashing - Consistent Hashing. Beautifully clear explanation of consistent hashing, a simple technique that allows you to add new caching servers to a cluster without re-hashing your keys and hence invalidating all of your caches.
deusx : Programmer’s Toolbox Part 3: Consistent Hashing | Spiteful.com - "Consistent hashing is a powerful idea for anyone building services that have to scale across a group of computers."
# copy
25 month ago
Simon Willison : Two data streams for a happy website - Two data streams for a happy website. Useful architectural concept for scaling: keep user-specific and generic data separate from the start, in recognition of their different caching and partitioning constraints.
deusx : Gojko Adzic » Two data streams for a happy website - "One of the most important architectural decisions that must be done early on in a scalable web site project is splitting the data flow into two streams: one that is user specific and one that is generic."
# copy
