Wednesday, November 26, 2014

Wireless Gamecard Deployment with Lessons Learned

I’m happiest when I’m able to get hands-on with a project and get some great tidbits of information in the process. I’m like a sponge, I soak it all in. In the last few months I have had the unique opportunity to work on a cool project between a vendor and one of their customers.

The purpose of lessons learned is to bring together insights gained during a project that can be usefully applied on future projects and it’s in that spirit that I pulled together this blog. Who knows – you might find some of this useful and be able to apply it in your own work. Besides, I think that it’s fun to talk about cool projects, and this is definitely a cool one :)

Please note: (For added effect, you may want to read this aloud at 1.5x normal speed.) While I may be a fan of the parties involved in this project, this blog is written independently of both companies. So, while it may seem like a paid endorsement, it isn't. I just tend to get excited about this stuff and sometimes that excitement comes across as marketing, which I kind of naturually gravitate to. I'm merely a nerd who enjoys hands-on work and seeing projects through to completion. That being said, I am a reseller/integrator of this gear, so please feel free to contact me to do this kind of work for you. If you are a vendor and you make a great product, I'll be happy to write about it when I deploy it too! :)

The Project: Provide Simultaneous Guest and Infrastructure Wi-Fi to 135 Video Game Card Readers & Thousands of fun-loving Customers

Every time I step into an arcade I feel like a kid again! It brings back memories of high energy and excitement. So imagine my giddiness at the prospect of working on a modern day gamecard deployment.

Remember when classic arcades took money? Then that turned into tokens. Then these turned into gamecards with wired card readers. Now, it's gamecards with wireless card readers! It's pretty awesome to see the transition taking place, not just in casinos, but something as fun as an arcade as well.

So what was I doing there? I was tasked with being the on-site resource to assist in the deployment of the multi-functional wireless network. The best part? I was handed a gaming card loaded with a huge amount of “tokens” to test out the machines. SCORE! The sounds, the sights, the colors, it's like Vegas for 10 year-olds. I hadn't realized the level of over-stimulation of these places until I walked the floor with every machine off, and then every machine powered up. It's nuts!

Somehow I managed to focus on the task at hand: how to securely get 135 machines on the wireless network, while still providing guest access, visitor loyalty, cloud management for 12 locations, and do it all cost-effectively? Simple, you deploy a wireless network from AirTight Networks.

So, what made this a great fit for this vendor, AirTight?

From the wireless connectivity side: Guest, Staff, and Device access are segmented across the APs keeping all the traffic flowing, while not allowing each of the independent networks access to each other. All of this can be controlled and configured regardless of location using the cloud control features of AirTight, while giving the end-user the ability to take advantage of profiles (location or equipment based) for all of the deployments nationally.

From the Guest Access perspective, the guests can be easily allowed access to the Wi-Fi network by using credentials from any of the major social media networks, a customized splash page, or a customized loyalty program. In addition, Retail Analytics and intelligence into the users on the network can be gathered and analyzed to help improve the customer experience through optimized business flow. Where do you put new games? Which bowling lanes are more active than the others? How much time is spent in each of the areas of the facility?

From the IoT / device / card reader perspective: The ability to configure the AirTight devices to accommodate such a large number of small-transaction clients (135 card readers firing up at the same time and downloading firmware via TFTP) simultaneously connected with big-data users in multiple groups. It was honestly a bit tricky at first, but in a testament to the agility and flexibility of AirTight's engineering and product resources, they worked to get the job done and the code rolled into a firmware release.

Bonus: That being said, here's a lesson learned!

When using TFTP, make sure you adjust the timeout accordingly for the transaction. In this case, we had over 100 devices connecting in, sucking down a huge file, and degrading the throughput for the "last in line" clients that needed their update. As the capacity dwindled, the response time of the TFTP server increased.

“If a packet gets lost in the network, the intended recipient will timeout and may retransmit their last packet (which may be data or an acknowledgment), thus causing the sender of the lost packet to retransmit that lost packet.” Source: Wikipedia.
Fewer packets coming in meant longer wait times to receive those packets, and in some cases waiting for the retry limit to hit (and timeout) to send another request for the data.

If the timeout was set at 120 or 300 seconds, then here's what would happen:

  • a few packets would come in, and
  • then the device would wait for 120 or 300 seconds, and 
  • then request more packets, 
  • resulting in up to 30 minutes to transfer a simple ~5mbps file. 

Block 1775 is the one to look at here. With timeout at 100 seconds, the timestamp shows 2146.996 when it started, and 2246.920 when it was acknowledged. This is part of the trace that showed us what was going on. The transfer to that unit took 25 minutes with a 100 second time out, and 3 minutes with a 3 second timeout.

When the timeout was set to 5 seconds, it took on average 3 minutes to download the file under full load and the same conditions. So (note to self), be weary of that TFTP timeout.

Overall, once the new firmware was in and the TFTP timeout was decreased, everything began to purr like a 3-day old kitten.

To watch the network react when the switch is thrown (or tapped in the case of their amazing automation system) and all the games come online was awesome. Combined with 3 school busses full of students, with who knows how many mobile devices, and the staff network to boot, it was a sight to be seen.

The people and technology at AirTight constantly keep me on my toes. I love being a part of and seeing them tackle different scenarios while helping their customers meet their needs, not just from a Wi-Fi viewpoint, but from an operations perspective, too. Very cool indeed!

Related Info:

Monday, November 17, 2014

Lessons Learned: Xirrus & the 2425H Array

I recently had to go fix up an install of 5 Xirrus 2425H arrays, and in the process of doing so have found some lessons learned that I felt I should share in case anyone gets stuck or needs some info on their deployment.
Not that this has anything to do with this post, but this thing was so bad ass its worth showing.


When I went to weatherproof the TNC connectors on the bottom of this unit, I learned something: bring patience ... or heat shrink tubing, a pair of scissors, and a heat gun. I opted for patience, as it was my first time doing this. The connectors on the bottom of the unit are grouped 4 together and leave very little room for weatherproofing and fingers. In retrospect, if I had some tubing and heat gun, it would've taken me 10 minutes tops instead of the 45 per device.
Also, as a bonus, there is lip created by the equipment cover on one side (the front face of the array) that makes it extra difficult to get your digits in there, so if you choose to go the route I did, get ready for some top-of-knuckle pain.

WDS: Definitely not a Xirrus strong suit (even though it's an awesome implementation of it)

Linking 2 arrays together on the Xirrus AOS is pretty damn simple using their WDS and a few dedicated IAPs: 
  1. Build an SSID for the bridge
  2. Define the array as the host
  3. Create a bridge client link with a check-box, SSID, MAC address, and password on the client IAP
  4. Coffee.

However, there is absolutely zero support for WDS in the XCS / Xirrus Cloud Management System, leaving you to configure it independently on each array / IAP.  Typically that wouldn't be a big deal, oh but it is. The XCS configuration gets pushed to the Xirrus array every time it updates from the cloud, overwriting the config on the array ... without WDS support .. which of course it disables when it writes the config. Luckily a week or two ago Xirrus decided to create "Read Only" configurations on XCS so that config settings wouldn't get overwritten on the array when it pinged the cloud... which is cool, however still pretty sucky because now you can't configure the arrays from the cloud, you can only monitor them .. sad trombone. 
So, lesson learned: if you're using WDS on Xirrus, try everything you can NOT to use the WDS (even though its awesome), including pricing out inexpensive bridges like Cambium ePMP or mesh gear so you don't have to lose the ability to config, control, and monitor your arrays from the cloud.

Learn the CLI

Lesson learned: It's awesome. Super quick, super awesome. SSH into the arrays using Putty. Golf clap to the CLI team.

Be Patient, It's Doing It's Thing

Xirrus arrays take about 3 minutes to reboot completely. For a minute or so they'll come up with as their default IP while they wait for the DHCP server to respond. You can access the array up until it gets a lease from DHCP using that default address, so if you need a quick fix for screwing something up, it's a cool trick.

Xircon = awesome

That simple little tool gives you a quick view of your network, the IP address of the arrays, and piece of mind knowing they are online. Great job on something simple yet totally usable.
Download it here using your partner portal login:

In closing..

All in all, I have enjoyed working on the gear (like a 6/10). The weatherproofing portion sucked big time. Followed up by the reboot-reset-reconnect-repeat issue with cloud updating, it has made for 4 days of on-site, unbillable service. So, heed the warnings on here and save some time and money. If I think of more, I'll post em. :)