Cookies Disclaimer

I agree Our site saves small pieces of text information (cookies) on your device in order to authenticate logins, deliver better content and provide statistical analysis. You can adjust your browser settings to prevent our site from using cookies, but doing so will prevent some aspects of the site from functioning properly.

Of Characters and Clouds

Bob
Getting right to the big news first, here's a screenshot of the first character to enter a version of Pathfinder Online with all of the game servers running in the cloud:

(Check out the screenshot on the blog post.)

Reaching this milestone is a huge step toward completing our cloud transition, so we thought this would be a good time to give you all a little more detail on how we're organizing the cloud effort, our progress so far, and what work remains to be done. Though we're not as far along as we'd hoped at this point, having run into a lot of complications along the way, we've made a great deal of progress and learned a lot of things that should prove helpful as we work through the remaining tasks and any additional complications that pop up.

To provide some context, we're dividing the cloud work into four big stages, each based on the four different server environments we move all new builds and content through as they go from initial development to final release. Those environments are known colloquially as Big Green Button, Chitterwood, Zog/Test and Barrow/Live. The new versions are being given the more helpfully descriptive (yet sadly mundane) names Dev (Development), Int (Internal), Stag (Staging) and Prod (Production), though we'll probably tend to use the original names interchangeably. Old habits die hard.

Dev is the minimum environment we need to run the game at all, and was previously called Big Green Button because you press a big green button on the helper app to start the servers up (technically, only the designers and artists use the helper app, programmers just use scripts, as the ancient coding gods intended). For Dev, the game's main applications and data are running on local machines instead of remote servers, with just enough server architecture (like database applications) running in the cloud to let everything talk to each other. This is helpful when developing new code and content because each developer can be running their own build and doesn't need to keep everything in sync with everyone else until they're ready to commit their changes. It was also a useful first step in moving to the cloud because we could focus on just getting those limited server functions working while we were getting accustomed to dealing with the cloud. Best not to tackle too many things at once. Even with that limited scope of work, Dev took a long time to pull together, but we got it working a few months ago and rolled quickly on to Int.

Int (Internal/Chitterwood) is the first environment where all the servers are running in the cloud, letting all the developers connect their clients to the same game world, and this is what we just got a rough version of working and took that screenshot in. So far we've been able to get all the game servers running, log in a character, and run around a bit. Right now Cole's the only one set up to run the servers and log in, so the next step is getting one of my machines set up so I can start testing a wider range of in-game activities, like running through multiple hexes, fighting escalations, crafting items, and all the other things players/settlements/companies can do. We expect most of them to run pretty much as they do on the current Chitterwood, but it's always possible we'll find a few surprises during testing.

Once we finish polishing Int up, it will essentially be a complete version of the game, just running a very small part of the map and with very minimal expectations for ease-of-use or stability. A few things will remain turned off, like Daily Maintenance and auto-billing, but the main systems will all be there for initial testing.

Getting to this first rough version of Int required climbing a steep learning curve, with Cole writing over a megabyte of notes on the way to adding DevOps to his ever-growing pile of developer hats. Meanwhile, I had to spend a lot of time crawling through billing spreadsheets and pricing documentation, since it turns out cloud work involves some accounting challenges on top of all the technical challenges. Fortunately, everything we've learned so far reinforces that we'll ultimately get everything running in the cloud, and seeing characters in a game world that's running completely in the cloud is one of the biggest proof-of-concept milestones toward that goal.

However, we did cut a lot of corners to get this rough version of Int working, so there's still a lot of cleanup work to do making everything stable and performant enough for the next stage, Stag (Staging/Zog/Test). That's when we'll be able to run a larger world and things will be dependable enough to start letting all you players take a look at it. To get there, we need to make things much more efficient so we can run a far larger map, optimize our server configurations to manage costs, and open things up so the game servers can talk to the wider internet instead of just our walled garden. Our current cloud setup also requires a lot of hand holding right now to get started up and stay running, so we need to create more flexible ways to manage our server configurations and tighten up the scripts that start, monitor, and stop all the various pieces of each development environment. We also got a cloud version of the website running so we could create and manage accounts on Int, but only in a rough debug mode, and there's still a lot of work to do before it can run in a proper production mode allowing the rest of you to access your own accounts and join us on Stag. In addition, we need to start sending the game's logs to a more permanent storage place so they don't get lost each time we need to make changes. Don't want to lose any of that useful data once we've got multiple people playing.

The final stage, Prod (Production/Barrow/Live), will largely involve polishing everything up based on what we find while testing everything on Stag, though there will also be a few unfinished pieces that aren't essential for Stag, like auto-billing and being able to make purchases at the Store. We'll also need to set up the process for shutting down our current Live servers, transferring all Live's final account and game data to the cloud, and starting back up in the cloud right where the physical servers left off.

It's hard to predict what other complications will arise as we finish off Int and move on to Stag/Prod, but we'll try to provide more updates along the way now that we've moved past initial cloud setup/configuration into getting individual parts of the game system working, tested, debugged and polished. We don't have a good ETA for when we'll get this all finished up, but hopefully we'll build up a better picture of that as we polish up Int and see more clearly which things are working well and which things need significantly more work. However, the destination is at least in sight now, a little blurry perhaps but still visible, and we thank you all for joining us on the winding path there.
malmuerta
Thanks for the update Bob!
Schedim
A truly herculean effort! Kudos to you both!
Schedim: Peddler and dealer in stuff easily transported, restless wandering the land of the River. Trying to find out how to reawaken the cult of Hanspur. To realise this ambition I created the company named Rats of Hanspur.
You can reach me on: pfo.schedim@gmail.com
Nails
Great news Bob!
harneloot
Your endurance and perseverance is admirable! Keep fighting the good fight!!
Xyzzy - gatherer, yeoman archer, swamp monster.
NightmareSr
Sounds vey promising Bob. Thanks to you and Cole for the tremendous work.
- Wandering gatherer (NightmareSr#2669 on discord)
– Cauchemar is a Greater Nightmare –
Maxen
Huzzah!
Bob
Over the last week, we managed to set things up so that I can get Int/Chitterwood started up in the cloud myself, then run the client on my computer and get into the game, allowing me to start testing a wider variety of systems. Much of the game works just fine (gathering resources, talking to trainers, accessing vaults, starting up escalations), but other things don't (can't open crafting doors, enemies aren't spawning, GM commands can't find characters on other hexes). The next trick is to start figuring out what's causing those issues, and to do that we're first working on accessing the cloud's version of the game logs.

In other news, we're just about ready to start making release candidates for the next update, adding the Combat Alchemist structures and a few other things. I'll post some preliminary release notes in a day or two, and with a little luck we may have something up on Zog later this week or early next week.
 
You must be logged into an enrolled account to post