Home

Interview: Jobe Makar from ElectroTank

Jobe is one of the guys that has been in the Flash game business "forever". He's an industry veteran that has created more than a hundred games, written/contributed to several books and made a living making Flash games. We talked to him about how he got started, Electrotank and the new Electroserver 4.

Interview by Jens C. Brynildsen [FM] When did you realize that you could live from doing just Flash?
[Jobe] I started with Flash when I was in college during Flash 3. About the time I was leaving school Flash 4 was released and was really spreading across the web like crazy. Then I started to see job postings and freelance postings. Companies would actually pay for this? Luckily for myself and for thousands of others, companies wanted what we could provide.

[FM] Electrotank is one of those names that have been around in the Flash community "forever". When and how did you get started?
[Jobe] In 2001 I was just leaving another company and was very interested in starting a company that focused on Flash games. I knew a very talented artist and "idea man", Robert Firebaugh, and I knew Mike Grundvig who was very interested in trying to build a socket-server for Flash games. The three of us were spread across the country and had never met in person, but we decided to give it a shot anyway. With a game programmer, artist, and server programmer we had all the people we needed to make almost anything we wanted. And we did!

[FM] Does that mean that you are still de-centralized?
[Jobe] Yes we are still spread across the country. Traditionally management hasn't been a problem. But in the last 2 years as we have taken on some pretty large projects, a new employee, and many contractors management definitely gotten more difficult. Since I have never worked in an office setting for more than a few days at a time I can't really say if it is easier or harder, but I suspect it is harder. Luckily we have a great team and some excellent contractors which allows for successful management. Today Electrotank focuses mostly on custom game development and our multiplayer server product. About 2 years ago companies started asking for multiplayer games. Now nearly everything we create for clients is multiplayer.

[FM] If I remember correctly, you first did games and then came the ElectroServer as a result of that?
[Jobe] From the public view of our company that was how it looked. The reality is that Mike and I were collaborating for months before creating Electrotank on some Flash / socket-server experiments. A very early version of ElectroServer (which didn't have a name yet) existed by the time we incorporated. Mini Golf - one of the first isometric Flash games on the web, was one of our first 2 or 3 games. This game has proven to be our most popular and is still actively pursued for corporate licensing today. It was our first successful multiplayer game. The initial popular of our Mini Golf game as a destination and as a licensing vehicle is what made us initially think about selling our socket-server product.

[FM] Yeah. I still love that game! It's a classic! It has a quite wide appeal since it's both "sports" and "puzzle"?
[Jobe] I think part of the appeal is that mini golf is a fun family friendly game that is easily recognizable. You can learn to play in moments and you can't lose.

[FM] How many games have you / ElectroTank created?
[Jobe] We don't keep an exact count, but the number is well more than 100. Before Electrotank formed I created many games. So personally I have been a part of probably around 200 Flash games. Custom game development is currently one of our primary company focuses. We actively develop games for many companies such as MTV, Mattel (Hot Wheels / Barbie), VH1, Disney, and Comedy Central.

[FM] How do you go about getting new clients? Is it all through the games you've already done or do you do "sales" in the traditional way to secure the clients you want?
[Jobe] We have been fortunate at Electrotank in that we have not had to approach companies to pursue sales. All of our business has been through word of mouth or other publicity that we have received though book writing, articles, speaking engagements, or awards. The only marketing that I can remember us doing is having a booth for ElectroServer at a couple of conferences a year.

[FM] You've contributed to quite a few Flash books?
[Jobe] Yes, over the years I have been lucky enough to have worked with Peachpit Press to publish several Flash books. The most popular were the Flash Game Design Demystified series and the ActionScript: Training from the Source series. Writing is definitely hard work, but it is also fulfulling. It forces you to understand every detail of what ever you are writing about. During the process of writing a chapter you can learn a lot. I am interested in writing still but find it difficult to find the time.

[FM] Is writing books worth the amount of work put into it?
[Jobe] That is a good question. In general book writing is not worth if financially. Even if the book makes back the advance and then some, it is still not much money as compared to what you can expect to make as programmer putting in those same hours. The worth, in my opinion, comes from how much you learn when writing the book, and the credibility you gain as an authority.

[FM] Electrotank apparently has such a huge variety of ways to make money. You offer game licensing, custom games, consulting, the ElectroServer, online games or downloadable coffe-break-games that are sold to end users... What is the main revenue-maker? (I assume it's not the server?)
[Jobe] You are right. Electrotank does have several sources of revenue. Over the years the main source has shifted from destination site related revenue (ads) to downloadable casual games, and now to custom game development. ElectroServer is a significant part of all of this since it is involved in nearly every project.

[FM] I assume you also make some money from support for the ElectroServer? How is that handled?
[Jobe] Up through ElectroServer 3 we have provided support at absolutely no cost to both license holders and to the public. As ElectroServer usage increases so have the support requests. At the moment we are in a losing situation on the support side. We have been forced to formalize the support agreement so that we can provide what we feel is reasoanble at no cost, and then charge for the time that our staff has to put in for more sustained support. With ElectroServer 4 developers will be able to navigate our site to find more thorough documentation than ever before, more examples, and tutorials and articles as they are written. Also, there is a developer forum where developers can discuss ElectroServer, and the Electrotank staff will contribute here as well as time permits. But for guaraneed support we provide four support packages ranging from Bronze level to Platinum level. These support packages are a combination of ticket-based support and providing an upgrade path to future major releases of ElectroServer.

[FM] ElectroServer 4 has been tested for up to 200K simultaneous users. That's an insane amount of users?
[Jobe] That is a ridiculous number! Our goal was to meet 100,000 simultaneous users. We were very excited when we met and surpassed that mark. In the future we will be performing more load tests where we expect to achieve much higher numbers when using the binary protocol. Because of the large numbers of simultaneous users that we support our server is ideal for massively multiplayer virtual worlds. The tremendously popular Webkinz World uses ElectroServer to achieve their large numbers. There are many virtual worlds currently in development that use ElectroServer as the connection layer. Most competing products do not support more than several thousand concurrent users.

[FM] What kind of hardware would such a massive pool of users require?
[Jobe] This is a common but tough question. It depends on what you are trying to do. Users that are primarily chatting, versus real-time games, versus audio/video are all totally different beasts. Developers can see our website or contact us directly to discuss what is needed for them. But in general, we are talking about commodity hardware with one bigger box used for the registry server. We aren't talking about tons of super expensive hardware.

[FM] What is the final limitation? Is it the software, bandwidth, the server BUS or other factors?
[Jobe] The final limitation in the current release of the server is the box used to power the registry server. The details of what causes a system to break down at hundreds of thousands of users is above my head (as a client developer), but it always come down to hardware. The server software can keep up, but is the hardware tuned enough to handle it?

[FM] How did you perform the test?
[Jobe] We built a flexible load testing application. The load tester itself is distributed across multiple machines. It runs multiple custom scenarios. For our big test we ran a combination of 3 scenarios: chat, avatar chat, and real-time game. Not only did we need to have many machines to fire up as gateway servers, but we needed to have quite a few available just to generate the load.

[FM] At the Flash On The Beach conference last year, you gave a talk on the challenges when creating a web based multiplayer game. I seem to remember that speed and latency of the users connection is the crux?
[Jobe] One of the most important goals is giving the user a fast, fun, and responsive game experience. Handling internet latency properly is a necessity to achieve this goal in real-time games. Most turn-based games naturally hide latency since users make a move once every several seconds. In a real-time game information is usually sent and received multiple times per second. A reasonable and common latency range is around 60-120ms. The upper end latency that you should attempt to program for is around 400-500ms. If someone has a higher latency than 400-500ms, or their latency is constantly jumping around, then this user will not have a good experience and the game can not be expected to provide it. Various predictive and smoothing techniques are used in these real-time games to allow for latencies in the ranges that I just mentioned.

[FM] How does ElectroServer help solve those problems?
[Jobe] ElectroServer does help in a few ways. But first let me mention that hiding latency is primarily the burden of the client. So most of the work in that case is done in ActionScript. However, ElectroServer does provide two major features that can help the game with displaying what it needs quickly. First, ElectroServer supports the concept of extensions. An extension is server-side code that gives ElectroServer more functionality by allowing code to be run scoped to a room/game (called plugins), run and scoped at the server level, or tied to server-based events such as logging in. This code is written using ActionScript or Java. Most real-time games will need a server-side plugin to manage the state of the game, or at least resolve important conflicts such as collision detection. Pushing important decisions to the server makes for easier development and a less buggy game.

The second important feature that ElectroServer provides is new to ElectroServer 4, Message Aggregation. Imagine a real-time tank battle game with 10 human tanks driving around and shooting at each other. Each tank may send position updates to the server 2 times per second and shoot at least once per second. That makes for 10 users * (2 position updates + 1 shot) = 30 server-bound messages per second - and this doesn't include health updates, collisions, or world-based events. Without knowing any better you might just mirror all of these messages back out to the users in the room. Well Flash has a problem with receiving a high packet rate. Our research and experience has shown that larger messgaes are not a problem, but high message rates are. If a Flash client receives more than around 5+ messages per second then you run the risk of what we call bottleneck latency. The Flash client can't process the messages fast enough so it ends up being delayed. So you end up with looks like higher latency when in fact the problem is in Flash. With ElectroServer, the developer has the option to turn on Message Agregation in the plugin and give it some settings. Let's say it was turned on with a rate of 200ms. That means that messages are sent no more frequently than once every 200ms, and and they are aggregated together. So instead of the Flash client receiving 30 inbound packets per second, it would receive 5, and each of those 5 would contain around 6 messages. In our testing we were able to go from 10 people in a tank game to 50 and not see a significant slow down due to message rates.

Jobe publishes a blog with lot's of neat tips for Flash game developers (dig past the recent ElectroServer news to find cool game related posts/code for pathfinding, crowd behavior, water physics, particles and much more). Jobe's colleagues Mike Grundvig and Robert Firebaugh (aka VectorKid) also have blogs worth reading if you do Game Development using Flash.