My personal Chaos Monkey

This is a repost of my Inspire9 blog post. It’s been over a year since my last post and I decided it was about bloody time for a new one.

About a year ago, I decided that life was too simple so I introduced the Chaos Monkey into my life.

The Chaos Monkey is a creation of Netflix which attacks it own systems by randomly shutting down services. As insane as it sounds, it is designed to ensure and prove their systems can gracefully tolerate any failure the AWS cloud can throw at them and build resilience in the process.

This time last year I didn’t have anything exciting happening. I had crawled away from my startup after a falling out with my partners. My other companies were earning a basic income for me but didn’t require much of my time and consulting generally sucks the life out of me. But the bills kept coming in and sitting around doing nothing wasn’t going to be practical for long.

When I first decided I wanted to move the family to the USA I had no clue I was releasing my Chaos Monkey. It seemed like a simple idea – follow my career-long dream of working in Silicon Valley by finding someone stupid enough to send me and my family overseas. Little did I know that I would spend the next 12 months consumed with this plan and that I would increase the family stress level to Def Con 4 by hopping on a roller coaster of success and failure that still hasn’t concluded its loops.

My first attempt was to acquire a job with a Melbourne company that wanted to open an office somewhere in the USA. I haven’t been an employee for 12 years so this was part distressing and part de-stressing. Within three weeks of starting I worked out how unlikely they were to ever be able to open an office in the USA and how even less likely I was to risk my family in their incapable hands.

My second attempt was to follow the (seemingly) sagely advice of Geoff McQueen who codified the process of setting up a US legal entity (in my case an LLC), offering oneself a job and then applying for an E-3 visa to accept that job. The E-3 visa application is a pretty simple process once it’s broken down into clear steps and really doesn’t require an immigration lawyer. Within about 3-4 weeks I had a company, a virtual mailroom, the authorisation to hire overseas workers and a visa application all prepared and ready to submit in person to the US consulate.

The interview at the consulate was going pretty well. They asked a few easy questions about me, my work and my company. I didn’t hide the fact that I was hiring myself and they didn’t make a major issue of it either. But they sent me away a 221(g) request for more information, specifically my Australian company’s “financial details”. I sent them P&L statements for the last 5 years. They asked for bank records. I sent them the last 60 days worth. Then a few more days later, without notice, my passport arrived without a visa in it and an accompanying 214(b) rejection.

It seems that the E-3 visa is not to be used for transferring yourself to the USA, rather for filling a role in a US company. The problem is this process is exactly what Geoff’s post suggests worked for him and many others. So, what went wrong? After speaking to a few people who also got rejected (where were they beforehand!?!?) the common thread is the Melbourne consulate rejects applications like this, the Sydney consulate accepts them. A few small changes might have made a difference but I will never know. I certainly should not have signed the paperwork (the letter of offer in particular) myself. That’s what highlighted to them that I was hiring myself.

The really nasty side-effect of this rejection is that from now on, whenever I am asked “Have you ever had a visa application rejected?” I have to answer yes.

I am now nearing the 12 month mark since I employed my first Chaos Monkey and I am still in Melbourne. I am working on attempt three by trying to lock down a job offer from a “real” US company so I can get an E-3 visa the standard way. My deadline is the end of this month. If I haven’t put the Chaos Monkey to bed by then, my wife will euthanise it. We are still married and we have learned a lot about patience, planning and stress management. A good example, is tell nobody about your plans until they are real.

 

For more information about the ins-and-outs of moving to the USA, have a look at our Wiki here.

Amazon Silk

http://amazonsilk.wordpress.com/

Ok, so I am breaking with tradition here and talking about a cloud service that really doesn’t fit within the definition of SaaS but that life sometimes. A quick disclaimer: I am proud to be an Amazon shareholder, AWS user and I think this will be my first Kindle.

Amazon has announced their latest Kindle Fire with a new browser called Silk. Silk’s claim to fame is that the work it performs of downloading and rendering web page is performed partially on the device and partially in the Amazon Web Services cloud infrastructure. The use of remote cloud grunt allows them to deliver a heavily optimised, cached and lower bandwidth rendering of the web pages you view using the massive and virtually limitless machine power they own and run. Only a few companies who are vertically integrated like Amazon, Google, Apple and Microsoft can offer services like this affordably, because they own or control much of the browser, device, network and servers.

From the Amazon Kindle Fire website, here is a quick summary of the advantages and technologies in use. I look forward to getting my hands on an early Fire and try Silk out for myself.

Revolutionary Cloud-Accelerated “Split Browser”

Modern websites are complex. A typical web page requires 80 files served from 13 different domains. This takes a regular browser hundreds of round trips, and adds seconds to page load times.Amazon Silk is different in a radical new way. When you use Silk, without thinking about it or doing anything explicit, you’re calling on the computing speed and power of the Amazon Web Services cloud (AWS). We’ve refactored and rebuilt the browser software stack to push pieces of the computation into the AWS cloud. This lets Silk do more work, more quickly, and all at once. We call this “split browser” architecture.Silk browser software resides both on Kindle Fire and on the massive server fleet that comprises the Amazon Elastic Compute Cloud (Amazon EC2). With each page request, Silk dynamically determines a division of labor between the mobile hardware and Amazon EC2 (i.e. which browser sub-components run where) that takes into consideration factors like network conditions, page complexity and the location of any cached content.

Shorter Transit Times

Amazon EC2 is always connected to the backbone of the Internet where round-trip latency is 5 milliseconds or less to most web sites rather than the 100 milliseconds that’s typical over wireless connections. AWS also has peering relationships with major internet service providers, and many top sites are hosted on EC2. This means that many web requests will never leave the extended infrastructure of AWS, reducing transit times to only a few milliseconds.

Computing Power in the Cloud

EC2 servers have massive computational power. On EC2, available CPU, storage, and available memory can be orders of magnitudes larger than on mobile devices. Silk uses the power and speed of the EC2 server fleet to retrieve all of the components of a website simultaneously, and delivers them to Kindle Fire in a single, fast stream. Transferring computing-intensive tasks to EC2 helps to conserve your Kindle Fire battery life.

Persistent Connections

A typical web request begins with resolving the domain names associated with the server and establishing a TCP connection to issue the http request. Establishing TCP connections for each request consumes time and resources that slow down traditional browsers. Silk keeps a persistent connection open to the backend server on the AWS cloud so that there is always a connection at the ready to start loading the next page. In addition, the Silk backend server keeps persistent connections open to the top sites on the web. This approach further reduces latency that would otherwise result from constantly establishing connections. Further, the connection between Silk and the backend infrastructure uses a pipelined, multiplexing protocol that can send all the content over a single connection.

Page Indexes

Traditional browsers must wait to receive the HTML file in order to begin downloading the other page assets. Silk is different because it learns these page characteristics automatically by aggregating the results of millions of page loads and maintaining this knowledge on EC2. While another browser might still be setting up a connection with the host server, Silk has already pushed content that it knows is associated with the page to Kindle Fire before the site has even instructed the browser where to find it.

Machine Learning

Finally, Silk leverages the collaborative filtering techniques and machine learning algorithms Amazon has built over the last 15 years to power features such as “customers who bought this also bought…” As Silk serves up millions of page views every day, it learns more about the individual sites it renders and where users go next. By observing the aggregate traffic patterns on various web sites, it refines its heuristics, allowing for accurate predictions of the next page request. For example, Silk might observe that 85 percent of visitors to a leading news site next click on that site’s top headline. With that knowledge, EC2 and Silk together make intelligent decisions about pre-pushing content to the Kindle Fire. As a result, the next page a Kindle Fire customer is likely to visit will already be available locally in the device cache, enabling instant rendering to the screen.

CloudFlare

http://www.cloudflare.com

‘Sorry about the break dear readers. I got knocked about by the flu this week and spent all of my blogging time curled up in bed with some lovely drugs and my misery. But I am back with a great service. CloudFlare was a bit of a surprise for me. My initial gut reaction was “another service leveraging the term ‘cloud’ to capture headlines” but something caught my eye and I decided I had little to lose.

CloudFlare is a website acceleration service. Essentially it is an automated reverse caching proxy and content delivery network (CDN). It takes over control of your DNS and serves up everything it can directly and passes through to your website what it cannot. Images, videos, CSS and Javascript files and any static HTML files will be served up from the “closest” web server CloudFlare operates and only what must be served from your server will be. This takes load of your server and network and even allows you to serve some content while you are offline.

For the moment, I choose to host my websites on a cheap and nasty shared hosting service. I host the most critical parts of my technology (such as DNS and email) on better quality service providers and generally speaking I am satisfied with the cost-value trade-off. Having said that I am quite aware that the website is a little laggy and could use some performance tweaking but I hadn’t got around to doing it.

So I was asked by a friend (Martin) to take a look at CloudFlare, which I did and in about 15 minutes time my site was setup and I just had to wait for the DNS change to propagate (about 24 hours or so). The result was not life changing but it was definitely noticeable. In particular the time from clicking a link to the finished result is “much faster” and thats all I wanted. As a bonus CloudFlare is now serving 50% of the data and 50% of the files instead of them coming off my server. I would imagine results will vary depending on what you are serving up, where you serve it from and where your audience is but thats the fun part.

You can pay for enhanced functionality and reporting but for a free service its a sweet offering.

I can’t find anything identical but there are a number of similar alternatives:

Wufoo

http://www.wufoo.com

I would love to say that I am such a great coder that I don’t need Wufoo and would never use it. But the reality is that Wufoo is such a well put together service that its easy to find value in what they do even for the best of developers. Wufoo is a form builder service – they create all sorts of forms for your users to fill out on your web page. The most common use for forms are:

  • Contact forms
  • Feedback forms
  • Surveys
  • Membership forms
But there are many many other uses for forms. They will even go so far as to take payments via your forms, with gateways such as Paypal. What Wufoo do is ensure your site professionally handles, filters, validates and manages your web forms so you can focus on your most critical parts of your web site. They will ensure your data is accurately collected and securely stored so you don’t have to worry. If you are updating or designing a new website consider implementing Wufoo in various places using one of their 80+ templates.

Wufoo were recently acquired by SurveyMonkey. Job well done guys. I hope you can continue to innovate in your new digs.

Alternatives:

DecideAlready

https://www.decidealready.com/

Pinch, punch first of the month. I thought today I would introduce you to a fun SaaS site. They are probably taking their site a little more seriously than I am but that’s life! DecideAlready is a group decision assistant. It allows a user to propose a question and a set of possible answers to a distributed group of individuals and have them vote on an outcome. There are three methods of decisions:

  • Simple – You pick one answer and assign one vote to it
  • Ranked – You rank all the answers from most to least preferred
  • Criteria – You provide them a list of factors to consider with a sliding scale of weight (this is their commercial offering)
So, this is not a life changing service. But it is a useful service and you would be surprised how many times we wish we had a product like this. A simple, easy, mostly free web service for solving an every day problem.
DecideAlready is a SproutBox portfolio company. They have a growing collection of cool companies offering simple but useful solutions such as (in order of what I find most interesting):
  • CheddarGetter – Awesome subscription management and billing system
  • Proposable – Web delivery of sales proposals
  • Squad  – Collaborative code editing
  • ScheduleThing – Online scheduling and reservations
Keep up the good work SproutBox.

Alternatives:

  • I can’t find any even though I know there must be some.
  • Post a comment to add to this list …

Notifo

http://notifo.com

Ok, so this is an app very close to my heart. I have been looking for a way of avoiding writing an app for delivering notifications to my users on http://www.mittosms.com. The cost of app development is still too high ($5,000+ for a basic application is the standard quotation I get) and I only need it for some basic communication with my clients. Notifo offers a managed third-party service with no-or-little cost of accessing the service and a trivial API to access all the functionality I need.

So, what does it offer?

For the developer (i.e. me):

  • Low cost and low stress technique for delivering a notification message (and a URL link) to a user
  • Multiple interfaces: API, Email, RSS
  • A free app for my users to download and use (it does other stuff as well!)

For the consumer (i.e. you?):

  • A free iPhone and Desktop app to use and abuse
  • A notification system for many different services: Twitter, RSS news feeds, Github, Nagios, Splunk, etc.
  • An awesomely simple one-click browser plugin for sending URLs to your phone
  • A basic P2P instant message system
So why do I love Notifo so much? It is the perfect integration model for startups like mine. I get free use of a critical service without having to code and distribute an app. It bridges gap perfectly between old world SaaS services and new world mobile SaaS apps. Then as my business grows I can afford to share a small amount of the love with them. Everyone has to make a living (except Google of course).

Alternatives:

Bit.ly

http://www.bit.ly/

Many of you will have seen a bit.ly URL but not thought too much about it. I have been using bit.ly for about a year now after year years of using tinyurl.com when I realised they were never going to improve their service. Bit.ly is probably the most popular URL shortening service around today (they perform 500+ shortens a second). Bit.ly shortens your long URL’s into really short ones that are easier to deal with (they don’t wrap in emails, they fit in SMS messages, etc).

Where biy.ly excels is the statistical analysis of your links. They will tell you how many people click on your link, how many people click on other people’s (version of the same) links as well as where, who, how, when, etc. They will even show you all the Twitter tweets which reference the same URL as you you. What I find of particular value is it keeps the archive of all your shortened URLs so you can go back and look at them in the future. There are a bundle of tools such as browser plugins and automatic converters but the real value is in the web site.

You can use the shortened links on any website or blog as well. For your efforts you get free analysis of every click (even when your link has been forwarded to someone else) for all time. However, I urge you to consider the environment next time you are sending an SMS or Email and shorten your URL’s. Then some time later go back to the bit.ly website and find out if your friends and family opened the links.

For a nice example …. imagine this as a tweet: “What will happen to http://bit.ly links if Gaddafi shuts down the Internet in Libya due to protests? – Quora http://bit.ly/oTkqVi” and then follow the statistics of this URL here: http://bitly.com/oTkqVi+

Alternatives: