Airbrakers–thank you for your patience with our major May migration. You’ve asked us to share details of our new infrastructure and this will be the first of many posts where we peel off one aspect of our deployment and share our approach.

Today we’re going to talk about our Frontend.

From 2008 through 2011, the entire Airbrake codebase consisted of a monolithic Ruby on Rails application. Over time this application became optimized to handle an ever-increasing volume of customers.

[ Graph showing increased numbers of exception processed by Airbrake. ]

At the end of 2011 we began a major backend rewrite that allowed the separation of the user-facing code (the “frontend”) and the mainly-api focusing code (the “backend”) that our 13 exception reporters use. Ruby, iOS, Coldfusion, Sinatra, .net, PHP, Android, Rack, Django, AppEngine, Redis, Merb, log4j, Chef, Erlang, Javascript, Java, and Drupal

Earlier this year we began testing our new backend with a subset of our users. At the same time, we looked for opportunities to allow us to make faster progress on our frontend. Giving us the ability to do things like WebHooks (in beta), GitHub Integration, Commenting, Better Debugging (Simulate Error) and more… (these all came to Airbrake in today’s release!)

Originally, we intended to host our new frontend (still Ruby on Rails) on Amazon EC2. We made this decision as our previous hosting facility was unable to keep up with our high-throughput needs–and our experience on Amazon in our other businesses (Exceptional and RightSignature) was positive in their ability to provision reliable service.

During our migration planning, we also brought up a staging server on Heroku (a PaaS that operates on Amazon). Amazingly–we had thought Airbrake to be sluggish when on our dedicated hosting provider’s servers; however, the Heroku app was significantly faster . It turns out that much of our performance was being wasted on calls within high-latency network calls to our support services (REDIS, Memcache, Legacy Mongo and our MySQL database). On Heroku (and therefore Amazon), these became low-latency, high-speed calls.

We spoke to the Heroku team and heard their commitment to making Heroku a platform that could handle Airbrake (at least our frontend) and we decided to use Heroku as one component of a robust deployment environment.

We have a commitment to demonstrating Heroku’s world class hosting environment by rocking Airbrake for their team and their most excellent users.

Oren Teich, Heroku COO.

 

With this support, we continued to focus on the components of our migration. We’ve added a pair of NGINX + HAProxy gateways on Amazon AWS XL instances (XL for network throughput). These serve Airbrake.io’s root domain. Requests to /notifier_api/v2/notices are routed to our new backend and the rest of the requests head onward to our frontend servers.

We launched two weeks ago with our own EC2 application pool as well as Heroku serving our application tier for our frontend. Unfortunately, our HAProxy configuration did not correctly account for session handling–and our ad-hoc attempts to remedy were causing more pain than relief–and during a fix attempt, we sent all frontend traffic to our Heroku application servers. We started to bump our Heroku dynos to match our usage and met demand at just over 75 dynos. We have identified a large number of our users have scripted access to /errors.xml and other popular endpoints. Having lots of these frequent requests necessitates a high dyno count. But, besides the dyno count, Heroku showed itself to be stably handling our application load (besides a few of these  that occurred as our initial 20 dynos were saturated).

We let this architecture settle in with our usage patterns and for the past two weeks we saw steadily faster-than-before access times for our users. We’ve been monitoring our support–and we have a few issues–which are mainly due to our new backend (next week!)–but our frontend has been blissfully consistent.

- Jonathan.

Today we’re happy to announce Github Issue integration.  Everyone will see this little button on the Error  Summary page.  Letting you send exceptions to Github Issues with one click.

This is version 0.1, and we welcome feedback on the integration in this Thread.  To set up Github integration you’ll need to add the Repro URL in the project settings.

The Github Issue linked to on the exception summary page.

The Octocat is a registered trademark of Github.. we made it out love for the service.

GITHUB®, the GITHUB® logo design, OCTOCAT® and the OCTOCAT® logo design are exclusive trademarks registered in the United States by GitHub, Inc. The OCTOCAT® design is the exclusive property of GitHub, Inc and has been federally registered with the United States Copyright Office. All rights reserved. No adaptation or use of any kind of any of our registered trademarks or copyrights, or any other contents of this website, is allowed without the express written permission of GitHub, Inc.

Monday saw 26 teams fight it out to become the “International Airbrake + Exceptional Charity Baggo Champion”.  Teams were from Airbrake + Exceptional, RedisToGo, Trigger.io, Intercom, OpenSignalMaps, Heroku, HipChat, Iron.io, EngineYard, CircleCI, CodeForAmerica,  Canonical, DolphinBrowser, singly.com, 42Floors, PivotalLabs, Pusher, VidCaster, UserVoice & Atlassian.

We had a double bracket, with some companies entering multiple teams. EngineYard had three teams, with their ring leader Dr Nic arriving early to get some practice in and to psych out the competition.

Each round was scored up to 11 points, a slight adjustment from the international cornhole rules.  Having 5 concurrent games helped speed up procedures.  The finals saw some epic matches with the current dodgeball champs Heroku vs Heroku in the semi-final. Ready for an Atlassian vs Heroku Final.  Scroll down to find out who won.

Read the rest of this entry »

After leaving San Francisco at 9pm, I finally arrived in Krakow at 11pm the next day.  This was my first trip to Poland, and I was excited to meet the people and take advantage of the strong dollar vs Zloty (the local currency).

A traditional Polish Breakfast Cheese, Ham, Cucumber, Bread and Fried eggs. This was an excellent way to start a day in Krakow. On my first day, I visited a few Airbrake customers; using the wonders of Intercom, I found 25 users, and after morning coffee I was chilling in their offices. Afterwards I made it to the Railsberry hackathon.

RailsBerry!

Railsberry is ‘a sweet European Rails Conference’ in Krakow, Poland.  It is put on by the crazy kids at Applicake and they have a passion for unicorns, rainbows, love and rails. It’s a two day conference with an eclectic range of talks.  Talks ranged from crowd funding, ruby project stats, coffeescript, NoSQL databases, and Enterprise SaaS to being a VIM ninja!  All talks were recorded and will be posted online.   My personal favorites were Eric Redmond’s NoSQL talk (as we switch to Riak) , Linda Liuka’s ‘Rails Girls’ talk for reminding everyone how awesome rails programming is, and Zach’s insight into rails stats.

There were a few awesome moments, such as 200 ballons going in the air [video], a 7ft unicorn, and some hilarious lighting talks.  I’m going to highlight all the awesomeness in another blog post, with 8 tips to throwing an exceptional conference.

Read the rest of this entry »

Earlier this month Airbrake processed its 8th billionth unique error, we’ve had peak days of 100 millon errors a day from our first wave of processing. Our rapidly expanding growth hit the limits of what Ruby could handle. After reviewing a few options node.js, Erlang, clojure we settled on Go as it provides the right mix of message queuing, systems programming and exceptional concurrency.

To deal with our ever expanding volume of data and to provide search and graphing we have migrated our datastore from Mongo to Riak. Expect to see graphs, search and smarter notifications in the coming months.

What you need to do?

The site should will keep working the same as normal, but if you have a restricted firewall you will need to add 75.101.128.85 to the list of outgoing IPs.

We will soon update our gem to send all traffic to http://api.airbrake.io, we will redirecting all old traffic to this new endpoint but we recommend all third parties update their libraries.

Issues

We’re currently importing old exceptions into our fresh new database and riak datastore. All errors should be present but expect all backtraces and “occurrence data” to be imported steadily over the next few days.

Photo from http://www.luckham.org/

WAZUP! It’s ya boy 50Cent here for the first Awesome Airbrake Cribs.  This new series  lets us visit and get a sneak peak into the coolest offices, hacker dens and brogrammers bedrooms.  We visited two awesome companies based in Kraków, Poland .

Read the rest of this entry »

We welcome GrubWithUs  to the eighth Airbraker in our “Awesome Airbrakers” series.

What does your App do?
We are taking social networking offline!

Users can have a fun time eating with new people in their favorite restaurants. They can browse users and meals on the site and find interesting people to Grub with!

Never eat alone!

Setup. Environment, Language, Tools, Etc.

• Rails
• Heroku
• Mongo
• Postgres
• Redis
• iphone

What do you like about Airbrake?
All exceptions in one place, figures out what errors belong to what deploy etc. Allows us to spend less time looking through our log files and more time coding our app.

How long have you been using Airbrake?
Since August 2011

How do you normally deal with Airbrake Exception reports?
Fix the bugs.

What could Airbrake do better?
Some of the JS errors have no backtrace and can be quite hard to figure out, also we can quickly fire out 1000s of JS errors from one dodgy ajax request that gets left running in a browser..

Favorite Music to Hack to.
Die Antwoord – Enter the Ninja

Anything Else?
Coffee lots of coffee.
@grubwithus
http://blog.grubwithus.com

In October of 2011 we bumped the Airbrake notifier for iOS and Mac to version 3.0. This near-complete rewrite of the product fixed a lot of known issues, reduced the number of dependencies, made it easier to install, and simplified the way in which developers interact with it. Since then we have been taking a lot of user feedback as well as contributions to the open source project found on github to fine tune its performance and give developers more control over its behavior.

The most significant change in recent months is the ability to control whether the notifier alerts users before posting crash reports. We got a lot of feedback about our requirement to notify users before data leaves the device and learned that a lot of apps wanted to handle this differently. Developers can now decide when and how their users are alerted about crash reports — if at all.

The other big change we made is the ability to decide which crash handlers are installed by the notifier. We realize that some apps may use different crash reporting mechanisms or even only want to log caught exceptions. The only way to perform that task in the past was to fork the project and make changes. Now it is as easy as passing a flag into the notifier.

With these changes your code might look something like this:

[ABNotifier startNotifierWithAPIKey: @"abc123" environmentName: ABNotifierAutomaticEnvironment useSSL:YES delegate:nil installExceptionHandler:YES installSignalHandler:NO displayUserPrompt:NO];

Note that while that is quite a mouth full we have been careful to preserve shorter methods for those who want to rely on the default functionality that the notifier provides.

We are always looking for ways to enhance the functionality of the notifier and the open source nature of the project is a big part of that. So if you have a great idea you are welcome to start a conversation about new features or fix a bug send us a pull request!

- Caleb Davenport

We welcome Alan from Treehouse to the seventh Airbraker in our “Awesome Airbrakers” series.

What does your App do?
We’re changing the world by teaching people how to do web development, web design, and iOS development.

Setup. Environment, Language, Tools, Etc.
Most of Treehouse is a Rails app, but we’ve also got some node.js in the mix. We do our front end scripting with CoffeeScript. We use MySQL for our data, and are just starting to play with redis for some features as well. We’re hosted at Rackspace right now, but are in the process of transitioning to EC2.

What do you like about Airbrake?
Airbrake’s super easy to work with. We get an email, we take a look at what happened, and we fix it. We haven’t checked out the javascript error reporting yet, but I’d really like to get that pulled into Treehouse.

How long have you been using Airbrake?
Since August 2011

How do you normally deal with Airbrake Exception reports?
Airbrake’s super easy to work with. We get an email, we take a look at what happened, and we fix it. We haven’t checked out the javascript error reporting yet, but I’d really like to get that pulled into Treehouse.

What could Airbrake do better?
I’d love to see more information about what parts of our code are causing the most exceptions over time. It’d also be great to be able to check and resolve multiple exceptions at once. We see some duplicates sometimes when the same code is causing exceptions in different spots.

Favorite Music to Hack to.

Right now I’ve been listening a ton to Rubber Soul by the Beatles, Broken Bells, and Rockin’ the Suburbs by Ben Folds.

Anything Else?
Our two developers are on Twitter as @commondream and @duwanis. We like coffee.

Airbrake and Exceptional welcome you to the first international charity exception throwing contest.

We are seeking all semi-pro exception throwers in the Bay Area for a spectacular evening of ‘baggo‘. A company must provide two throwers and teams will compete in sudden death mode. Spectators can watch the world’s finest bag toss players play it out to death.

How it works / Enter your team.

1. A company must submit two team members. Each team member must get a ticket, picking from above. ($50 total for a company, or $100 for a 418). Once both team members have tickets, e-mail ben@airbrake.io and I’ll add you to the current teams list at the bottom of this page.
2. Teams will play to the death in a sudden death mode.
3. Games will commence in the basement of the Storek Building at 6:00pm sharp. Food will be provided.
4. The champions will receive a pair of Exceptional Kicks.

Charity

All money collected will be donated to

1. http://camara.ie/ an epic charity that is dedicated to using technology to deliver education more effectively to disadvantaged communities in Africa and Ireland.
2. https://codeforamerica.com an epic non-profit helping government work better for everyone.

Sponsor our Bean Bags?

For a $250 donation we will make custom bean bags for your company using the wonders of Spoonflour and the handicraft of Ben. Please contact ben@airbrake.io for more details.

Current Teams:

• Airbrake + Exceptional
• RedisToGo
• Trigger.io
• Intercom
• Iron.io