Castro blog

The latest news for podcast lovers

Back to list

Castro 2.0: The Servers

Posted by Padraig on Aug 09, 2016

Our new app, Castro 2.0 is coming to the App Store on Monday, August 15th 2016 at 9am PT. The following is a developer post about how we built the server backend that powers the app.

Tentacles

A primary goal for Castro 2 is to enable users to subscribe to a much wider range of podcasts than before, so it was important to reduce the battery and bandwidth cost of subscribing to a lot of shows. Castro 2 is powered by a backend server cluster that we call “Tentacles”. Its job is to regularly scan every single podcast rss feed to find new episodes or metadata changes (e.g. new artwork or spelling corrections). When it finds them, it notifies Castro users who subscribe to that podcast. This takes the burden of doing this work away from the client app that runs on user’s iPhones.

Tentacles also allows us to host episode share pages which open directly in the app for users, allowing them to queue the episode without needing to subscribe. We also built a web based player as a fallback for users who don’t have Castro 2 yet.

Designing the Server

I’ve worked on a few backend server systems in the past. Once for a company named Neteller, which used a language called ColdFusion™. That was the job that taught me that good code has very little correlation with business success (they were very successful). Later I helped to build a system using proprietary and very expensive tools from a company called TIBCO.

But that was all before 2005. I haven’t done much of this kind of work since then, so when it was time to build Tentacles, I was completely ill equipped to build a backend server for Castro.

Luckily for Oisin and I, a good friend of mine in Vancouver, Ryan McCuaig, does know how to do this stuff. We negotiated a contract with Ryan to set up a “walking skeleton”, which meant getting the big components in place for me to flesh out the details later.

It worked out fantastically; it has been absolutely great to work with Ryan. The server project quickly transformed from one I had been dreading into something I was excited to wake up and work on.

Server-side code has its own challenges, but the ease of making a change and instantly deploying it is a special joy to an iOS app developer like me. If some podcast details are parsed incorrectly in Castro 2, we’ll be able to fix them for all users in just few minutes. Back when we released Castro 1, fixing a bug like this and releasing it was, at best, a one week process and users who didn’t auto-update might not see the fix at all.

The Stack

Tentacles is composed of a number of servers:

  • A load balancing server running nginx, receives requests and routes them to one of the available web servers. This means we can deploy changes to servers without any downtime as well as easily scaling up the number of web servers.
  • A few Rails web servers to host the API.
  • A few worker servers running Rails with Sidekiq, constantly refreshing all podcast feeds.
  • A Postgres database server (and another one running as a standby)
  • A Redis server, mainly for caching API responses.
  • A statistics server, running carbon/graphite to allow us to monitor the cluster.

Deployment

One of the early decisions that Ryan made was to build each server in a scripted, repeatable way. We don’t spend time hand-tuning a given server to get it just right — all configuration, tweaks, everything, happens this way. If one server gets weird, we can kill it and build a new instance in a few minutes. This makes scaling up or down fast and removes so much wasted dev-ops time that I will now never build a server any other way.

Our build scripts are all written using Ansible 2.0 but I’m not particularly attached to Ansible over any of the other options. The important part is not to waste time with artisanal hand-crafted servers. Using build scripts is similar to using source control in that there’s a bit of up-front learning to do, but the payoff is huge.

Coding

I do all Tentacles development using Sublime. Ryan uses some kind of 28-dimensional VIM matrix interface, that I initially mocked but now quietly fear. I resisted Sublime initially, but I wasn’t able to find a great IDE for server side development. RubyMine is as close as it gets; I tried it for a while but it’s really expensive and painfully non-native. I wanted to use Coda but its so well tuned for direct HTML/CSS design and instant previewing that writing a backend API seemed like the wrong project for it. The amount of plugins and configuration (by editing a JSON file, wtf) required to get Sublime set up just right offends my precious sensibilities, but once you get it done it’s a fantastic tool.

In learning to write and deploy server code, I’ve noticed a lot of differences from the app development world. The biggest one is just how ingrained and well thought out the idea of automated testing is. I’m indebted to Ryan for helping me pick this stuff up so quickly.

Development Environment

I don’t really want to install and run a database, redis, web servers, etc. on my Mac which I use for other things too.

We use Vagrant to build a linux virtual machine development environment using the same Ansible scripts I mentioned above. I can share the folder that has the project code from my Mac to the VM. I still get to edit in Sublime, but everything runs on linux. This means that we all run the same environment which eliminates “well it works for me” issues. If I mess it up somehow, I can just destroy the VM and rebuild it in about 20 minutes. All of the server components are nicely boxed off this way, so once I shut down the VM, I’m no longer killing my Macbook battery. I’m also automatically running the right version of Ruby and can’t be screwed by an OS X update.

Vagrant development hasn’t been quite as smooth as it might have been. There have always been a few lingering issues where changes in one place have unexpected consequences in another. I end up debugging that for an hour once in a while, but in principle the approach is right.

Conclusion

I’m really happy to have learned how to build, develop and manage a modern server cluster. It’s a great skill to have and opens up a lot of opportunities for future app ideas for Castro and beyond.

Our next challenge will be to launch the service to thousands of users next Monday…

Sign up to stay up to date

Get the latest news on Castro product updates and new features.