Wednesday, November 26, 2014

Wireless Gamecard Deployment with Lessons Learned

I’m happiest when I’m able to get hands-on with a project and get some great tidbits of information in the process. I’m like a sponge, I soak it all in. In the last few months I have had the unique opportunity to work on a cool project between a vendor and one of their customers.

The purpose of lessons learned is to bring together insights gained during a project that can be usefully applied on future projects and it’s in that spirit that I pulled together this blog. Who knows – you might find some of this useful and be able to apply it in your own work. Besides, I think that it’s fun to talk about cool projects, and this is definitely a cool one :)

Please note: (For added effect, you may want to read this aloud at 1.5x normal speed.) While I may be a fan of the parties involved in this project, this blog is written independently of both companies. So, while it may seem like a paid endorsement, it isn't. I just tend to get excited about this stuff and sometimes that excitement comes across as marketing, which I kind of naturually gravitate to. I'm merely a nerd who enjoys hands-on work and seeing projects through to completion. That being said, I am a reseller/integrator of this gear, so please feel free to contact me to do this kind of work for you. If you are a vendor and you make a great product, I'll be happy to write about it when I deploy it too! :)

The Project: Provide Simultaneous Guest and Infrastructure Wi-Fi to 135 Video Game Card Readers & Thousands of fun-loving Customers

Every time I step into an arcade I feel like a kid again! It brings back memories of high energy and excitement. So imagine my giddiness at the prospect of working on a modern day gamecard deployment.

Remember when classic arcades took money? Then that turned into tokens. Then these turned into gamecards with wired card readers. Now, it's gamecards with wireless card readers! It's pretty awesome to see the transition taking place, not just in casinos, but something as fun as an arcade as well.

So what was I doing there? I was tasked with being the on-site resource to assist in the deployment of the multi-functional wireless network. The best part? I was handed a gaming card loaded with a huge amount of “tokens” to test out the machines. SCORE! The sounds, the sights, the colors, it's like Vegas for 10 year-olds. I hadn't realized the level of over-stimulation of these places until I walked the floor with every machine off, and then every machine powered up. It's nuts!

Somehow I managed to focus on the task at hand: how to securely get 135 machines on the wireless network, while still providing guest access, visitor loyalty, cloud management for 12 locations, and do it all cost-effectively? Simple, you deploy a wireless network from AirTight Networks.

So, what made this a great fit for this vendor, AirTight?

From the wireless connectivity side: Guest, Staff, and Device access are segmented across the APs keeping all the traffic flowing, while not allowing each of the independent networks access to each other. All of this can be controlled and configured regardless of location using the cloud control features of AirTight, while giving the end-user the ability to take advantage of profiles (location or equipment based) for all of the deployments nationally.

From the Guest Access perspective, the guests can be easily allowed access to the Wi-Fi network by using credentials from any of the major social media networks, a customized splash page, or a customized loyalty program. In addition, Retail Analytics and intelligence into the users on the network can be gathered and analyzed to help improve the customer experience through optimized business flow. Where do you put new games? Which bowling lanes are more active than the others? How much time is spent in each of the areas of the facility?

From the IoT / device / card reader perspective: The ability to configure the AirTight devices to accommodate such a large number of small-transaction clients (135 card readers firing up at the same time and downloading firmware via TFTP) simultaneously connected with big-data users in multiple groups. It was honestly a bit tricky at first, but in a testament to the agility and flexibility of AirTight's engineering and product resources, they worked to get the job done and the code rolled into a firmware release.

Bonus: That being said, here's a lesson learned!

When using TFTP, make sure you adjust the timeout accordingly for the transaction. In this case, we had over 100 devices connecting in, sucking down a huge file, and degrading the throughput for the "last in line" clients that needed their update. As the capacity dwindled, the response time of the TFTP server increased.

“If a packet gets lost in the network, the intended recipient will timeout and may retransmit their last packet (which may be data or an acknowledgment), thus causing the sender of the lost packet to retransmit that lost packet.” Source: Wikipedia.
Fewer packets coming in meant longer wait times to receive those packets, and in some cases waiting for the retry limit to hit (and timeout) to send another request for the data.

If the timeout was set at 120 or 300 seconds, then here's what would happen:

  • a few packets would come in, and
  • then the device would wait for 120 or 300 seconds, and 
  • then request more packets, 
  • resulting in up to 30 minutes to transfer a simple ~5mbps file. 

Block 1775 is the one to look at here. With timeout at 100 seconds, the timestamp shows 2146.996 when it started, and 2246.920 when it was acknowledged. This is part of the trace that showed us what was going on. The transfer to that unit took 25 minutes with a 100 second time out, and 3 minutes with a 3 second timeout.

When the timeout was set to 5 seconds, it took on average 3 minutes to download the file under full load and the same conditions. So (note to self), be weary of that TFTP timeout.

Overall, once the new firmware was in and the TFTP timeout was decreased, everything began to purr like a 3-day old kitten.

To watch the network react when the switch is thrown (or tapped in the case of their amazing automation system) and all the games come online was awesome. Combined with 3 school busses full of students, with who knows how many mobile devices, and the staff network to boot, it was a sight to be seen.

The people and technology at AirTight constantly keep me on my toes. I love being a part of and seeing them tackle different scenarios while helping their customers meet their needs, not just from a Wi-Fi viewpoint, but from an operations perspective, too. Very cool indeed!

Related Info:

Monday, November 17, 2014

Lessons Learned: Xirrus & the 2425H Array

I recently had to go fix up an install of 5 Xirrus 2425H arrays, and in the process of doing so have found some lessons learned that I felt I should share in case anyone gets stuck or needs some info on their deployment.
Not that this has anything to do with this post, but this thing was so bad ass its worth showing.


When I went to weatherproof the TNC connectors on the bottom of this unit, I learned something: bring patience ... or heat shrink tubing, a pair of scissors, and a heat gun. I opted for patience, as it was my first time doing this. The connectors on the bottom of the unit are grouped 4 together and leave very little room for weatherproofing and fingers. In retrospect, if I had some tubing and heat gun, it would've taken me 10 minutes tops instead of the 45 per device.
Also, as a bonus, there is lip created by the equipment cover on one side (the front face of the array) that makes it extra difficult to get your digits in there, so if you choose to go the route I did, get ready for some top-of-knuckle pain.

WDS: Definitely not a Xirrus strong suit (even though it's an awesome implementation of it)

Linking 2 arrays together on the Xirrus AOS is pretty damn simple using their WDS and a few dedicated IAPs: 
  1. Build an SSID for the bridge
  2. Define the array as the host
  3. Create a bridge client link with a check-box, SSID, MAC address, and password on the client IAP
  4. Coffee.

However, there is absolutely zero support for WDS in the XCS / Xirrus Cloud Management System, leaving you to configure it independently on each array / IAP.  Typically that wouldn't be a big deal, oh but it is. The XCS configuration gets pushed to the Xirrus array every time it updates from the cloud, overwriting the config on the array ... without WDS support .. which of course it disables when it writes the config. Luckily a week or two ago Xirrus decided to create "Read Only" configurations on XCS so that config settings wouldn't get overwritten on the array when it pinged the cloud... which is cool, however still pretty sucky because now you can't configure the arrays from the cloud, you can only monitor them .. sad trombone. 
So, lesson learned: if you're using WDS on Xirrus, try everything you can NOT to use the WDS (even though its awesome), including pricing out inexpensive bridges like Cambium ePMP or mesh gear so you don't have to lose the ability to config, control, and monitor your arrays from the cloud.

Learn the CLI

Lesson learned: It's awesome. Super quick, super awesome. SSH into the arrays using Putty. Golf clap to the CLI team.

Be Patient, It's Doing It's Thing

Xirrus arrays take about 3 minutes to reboot completely. For a minute or so they'll come up with as their default IP while they wait for the DHCP server to respond. You can access the array up until it gets a lease from DHCP using that default address, so if you need a quick fix for screwing something up, it's a cool trick.

Xircon = awesome

That simple little tool gives you a quick view of your network, the IP address of the arrays, and piece of mind knowing they are online. Great job on something simple yet totally usable.
Download it here using your partner portal login:

In closing..

All in all, I have enjoyed working on the gear (like a 6/10). The weatherproofing portion sucked big time. Followed up by the reboot-reset-reconnect-repeat issue with cloud updating, it has made for 4 days of on-site, unbillable service. So, heed the warnings on here and save some time and money. If I think of more, I'll post em. :)

Friday, October 24, 2014

The Holy Grail for WISPs? What could it be?

This was just posted to the WISPA general list. I only copypasta'd here because I think it's awesome to build up campaigns like this. I just hope that whatever it is, it lives up to the hype, and delivers.  What do you think you could be?

For WISPs, is the "Holy Grail" coming soon?

What if there was a technology you could use that...

- Could realistically generate $35,000/mo or more in revenue from a single tower site, leaving your controller with a perpetual goofy grin?
- Was able to connect in the real world over 350 subscribers per tower with a service package of 20 Mbps DL/5 Mbps UL without overtaxing the system and do it with just 3 sectors?
- Not only can reasonably connect well over 100 subscribers per sector, but we can prove it by logging into a system running live with hundreds of sector examples and tens of thousands of live subscribers, and can show you utilization across all time on any one to prove it?
- Allowed you maybe even double the above over time without another climb or a dime more?
- Enabled provable NLOS service in scale - even in 3.65 GHz - far better in range, capacity and stability than anything you've ever seen in 900 MHz?
- Included on every SM a built-in SIP client and POTS jacks?
- Featured SMs that had an optional built-in Wi-Fi AP fully manageable over the network to make your service more sticky?
- Had base stations that will eventually be able to configure and optimize themselves in real time without human intervention? Climb, mount, turn on, climb down.
- Also had SM options with dual N type connectors?
- Could manage the interference environment in even just 50 MHz so well we could show you a place with over 300 base stations across 600 square miles and over 20,000 connections, all operating as engineered?
- Permits your technicians to sleep soundly at night, ...and your competitors to have nightmares?

What would something like that be worth to your business?

It is coming, and soon. Do you want to know more? Send us a mail and we'll keep you in the loop offlist.

...oh, and let us know if you think we are just spinning fairy tales. We respect and love skeptics. We expect they'll be among our best new customers.


Saturday, October 18, 2014

Connected Cars and Cameras on Dashboards

So I am definitely not the authority when it comes to connected cars. I'm pretty good at building wireless networks for municipalities, enterprises and businesses, but when it comes to the carrier stuff and building out a national network that's not really my expertise. The reason I preface this blog post with that is because I'm curious about the connected car.

On a recent drive from Houston, Texas down my hometown of McAllen, I saw a Ford Mustang with GoPro attached on the dashboard and two more cameras on the front of the car above the headlights. As anyone who's ever surfed the Internet knows, that is not an uncommon sight in Russia or any Eastern European country. Why? I dunno. You can scour YouTube and find videos from dashboard cams from all over Eastern Europe. However that doesn't necessarily seem the case of United States.

So what I started thinking about was this: if that GoPro was connected via Wi-Fi to the vehicle, and then out to the web, what opportunity would that open up for someone like, say, the insurance companies? Take that Flo. As more connected cars get introduced into the marketplace, specifically talking about GM's announcement to have 33 models all with 4G service, what kind of applications are we going to find for Broadband in the vehicle?

At CTIA's Super Mobility Week this year there were tons of examples of what you could do with the connected car. The CTO of Tesla motors talked about sending suspension updates down to the vehicle so that the car ran smoother and more efficiently for example. But I wonder how other people are really going to start using this type of connected technology to do things with cars that we haven't even thought of yet; and again, I am not a voice from the connected car manufacturing community, or a wireless carrier that serves up LTE across the continent, but I'm just a nerd trying to figure out different ways that I would use broadband.

With specific regard to the concept that I was just talking about, I can't imagine how awesome that would be for the insurance company to have cameras mounted into the bumpers an sides of the vehicle with a small DVR in the vehicle. It could not only be recording video and imagery, letting you review what happened, when it happened, how it happened, but when combined with all of the other sensor data on the vehicle including GPS imagine how that's going to make life easier for insurance adjuster?! If you're in an accident they don't have to wait to determine who's fault it was based on the police report, they can review the on-board sensors and on-board video to immediately figure out who was involved and how the accident took place.
When some developer of a connected "black box" for vehicles starts to push this out the insurance agent, I would not be surprised if it became standard equipment on every vehicle that goes off the lot.

I have a friend that sells used cars and he does something similar, not only to track his cars but also to have a killswitch so that when a payment is not made they can disable the car, grab the GPS location and go out to the site to pick up the vehicle. There was just an article published about this and circulated nationally where both sides of the argument came out in force.

If vehicles now have broadband connectivity built into them and it can be leveraged to support things like that on a much larger scale, as well as all of the sensor data, pushing firmware updates to braking system, etc. I think it could definitely impact that entire industry in ways we have yet to see or even imagine. When wireless took off, first with wireless cable, then the WISPs, then the muni broadband phase, and the overall proliferation into everyone's home and offices we saw this happen. Every PowerPoint slide had the "If you build it they will come" tagline. We saw connectivity being used for so many different things that we never thought of ... for whatever reason. Nest? Really? But what a great application. Imagine this "new" frontier. It's gonna be pretty exciting!

Anyhow, those are just my random thought about this subject, but I think we should really start to think about where this is going to lead the car industry not just from a connectivity standpoint and ability to watch SpongeBob SquarePants on a road trip, but how this is going to benefit every different level of business in automotive industry. I'm looking forward to the innovation.

As with all of my posts, thank you for taking the time to read this random string of thoughts and I hope you have a great week!

Sunday, October 12, 2014

WiFi Calling and VoLTE Network Readiness: Tidal wave or splash in a puddle?

I was out on an install recently and speaking with a group that runs the connectivity and IT service for a group of libraries in a City. We were discussing firewalls, application awareness, and eventually good ol' Wi-Fi. When someone brought up the iPhone 6, we began discussing VoLTE and Wi-Fi calling. The conversation that ensued had us all looking around the room at each other and realizing we weren't the only ones that had stumbled across this uh-oh moment.
But is it really that big of a deal?
I want to think this out via blog post..

Number of Devices on the Network
So, let's say everyone puts HD voice, or Wi-Fi Calling, or VoLTE on their phone. Now wherever they have a Wi-Fi connection, their calls are routed via the Wi-Fi chipset and across your APs and your network. I think this part of the conversation has more impact on your network than the next part. Right now, Wi-Fi adoption rates at home, work, and play are in a pretty high spot, but I think it can and will go higher. If carriers introduce a Wi-Fi calling plan that doesn't eat away at your minutes or they continue to drop the bar on data caps, or if mobile net neutrality doesn't come through, people will be loving that Wi-Fi at work, school, library, retail store, etc.

So many things have contributed to the increased use in Wi-Fi over time that one more item, like this push to packetize voice into VoLTE or Wi-Fi calling, is just going to increase that, and that's going to increase the load on your network. More devices require an infrastructure that can handle, more devices. As is right now, too many networks are stressing their WLAN equipment in an effort to offer wireless services to everyone, just think about how awesome it will be once (to use a 2014 millennial generation word of the year) literally everyone gets on that network? Awesome, right?

That library customer that I was visiting now becomes the carrier .. not really, but, what's the difference? If I am a Verizon customer having problems completing a call at your library because your Wi-Fi network can't handle the number of devices or the traffic on the network, who gets the support call? You guessed it, whoever's closest. That means the person working the help desk gets the question "is your wifi working, I can't make a call" which prompts he / she to call you, which leads to.. well, you get the picture. This might not happen all of the time, but with more of a dependence on Wi-Fi networks from mobile devices, tablets, phablets, etc. it can and will become an issue.

So what do you do?
First of all, make sure your wired network infrastructure is capable of handling these high-speed connections to your wireless network. Think about your cabling; is it capable of running GigE across it? How about PoE and PoE+? It might be time for an infrastructure upgrade in the wiring department. On that note: Go big or go home. If you're looking for a great and informative free lunch, this is totally a great time to call your disti and ask to have the cable rep take you to Sizzler and drop some science on you. Trust me, you'll get a kick out of it. Just say: "What's the difference between your cable and the stuff from Home Depot? Cables is cables, right?" Watch them stop chewing immediately.

Now, what about those pesky switches? These aren't gonna be 100Mbps ports we're talking about after all. What good is all that fancy MU-MIMO and open airspace in the 5GHz spectrum if you're dumping it into a 100Mbps port? I'm not saying to get knee-deep in this, but if you do, why not check out some of these. I just signed up as an Arista dealer, and man, these things blow me away!

Now that you have a kick-butt foundation, figure out if your wireless infrastructure is capable of supporting the number of devices that will be sitting on it in the very near future. It might be, it might not be. My rough estimate is figuring on 30 devices per AP, for now (granted, that is a very rough estimate). If you think about your users, some may have 2 devices, that may give you around 20 users per APs. If you think there will be more than 30 devices, figure out the best way to scale up or engineer a solution for it. I'm not talking about simply adding more access points and increasing your noise-floor, I'm saying figure out the best design for your network that can offload some of those users onto more strategic access points, take advantage of technologies like beamforming, band-steering, load balancing, etc. This will more than likely require some professional engineering, so don't skimp.

Now, if you haven't upgraded to N or AC, start to think about that, like yesterday. Freeing up your spectrum on 2.4GHz and adding capacity on 5 GHz is a must. With the iPhone 6 and all future devices supporting 802.11ac, give those users a place to take advantage of that technology. Start phasing it in if you have to, but get the ball rolling.

Anyhow, some of this might hit home with you, some of it might not. Either way, whether you want to pay attention to this blog post or not, know that it is coming. You might not have to order a ton of Arista 7150's but at the end of the day your infrastructure will be used to place and receive calls from Joe-public. When that happens it's probably better to be on the side of supporting it without a headache than scrambling to figure out what to do. Keep in mind, whichever situation you're in, feel free to call and I'll sell ya stuff and maybe even take you to Sizzler. :)

Have a great week!

Wednesday, October 1, 2014

Installing Ruckus FlexMaster on HyperV running CentOS 7

I'm only posting this because it has been my journey for the last 2 days, and I want to share the secret sauce that worked for me. PS, in the release notes FlexMaster is only supposed to run on RedHat Enterprise Linux and CentOS 6.3 or 6.5, so take that as a warning that while this install works, it is not recommended or supported by Ruckus .. I'm just doing it because I like to be difficult. :)
  1. Create a New Virtual Machine
  2. Name it what you will
  3. Make sure to give it plenty of memory
  4. Choose the right NIC (Important: if you are using CentOS 6.3 or 6.5 you have to use the Legacy Adapter Option after setup of the VM completes or else CentOS won't see your card)
  5. Plenty of space for storage
  6. Choose the right image
  7. Hit the Finish Button
  8. Connect to your new VM and start that bad boy up.
  9. Choose Install from the menu.
  10. Now you have the "Installation Summary" page. Under "Software Selection", you're gonna want to choose a couple of add-ons that are needed by the FlexMaster install.

  11. I used the "Basic Web Server" config and added-on: PHP Support, Perl for Web, Compatibility Libraries and Development Tools
Once you're done with that first part, the install starts and your VM comes up. Login with root. Here's what I did that worked:

  1. Make sure you have the latest ISO of the FlexMaster software on-hand, and downloaded from the Ruckus Support website.
  2. Fire up an FTP server on a machine, the HyperV server, or somewhere on the local network. Then, use wget to transfer the file:
  3. # wget ftp://IPADDRESS/FM_Software.iso
  4. create a mount space using:
  5. # mkdir /mnt/flexmasterinstall
  6. Mount the iso of the flexmaster software to that location:
  7. # mount -o loop /FM_Software.iso /mnt/flexmasterinstall
  8. This part was pretty important: copy the files from the iso to a directory on the machine:
    # cp -r /mnt/flexmasterinstall /opt/flexmasterinstall
  9. Once that's done:
  10. # umount /mnt/flexmasterinstall
    # rmdir /mnt/flexmasterinstall
  11. From here, get into the dirctory and run the install command:
    # cd /opt/flexmasterinstall
    # ./
  12. Let that thing go and answer the questions.
  13. Once it's done, don't forget to do this: open up that firewall!
  14. Here's how to open the firewall for http and https (kinda important to do https considering ZoneDirectors require an HTTPS option to communicate with FlexMaster).. and don't forget to restart.
    # firewall-cmd --permanent --add-service=http
    # firewall-cmd --permanant --add-service=https
    # systemctl restart firewalld

That's it! Go to the IP address of the server and you should be good to go!
Please note, this might have some errors, cus I'm on 2 monsters and some Limonata Sanpellegrino.

Monday, September 22, 2014

Smart Dining & Dining Smart: Technology in the Restaurant Industry

An area that we have been doing some amazing things in, and winning some sweet awards, has been the restaurant industry. Like the majority of the people reading this I spend quite a bit of time sitting on the other side of the table of this industry. From coffee shops and ice cream parlors, to white table cloths, I'm always doing a little bit more than looking at the menu and trying to figure out what to eat.

As a nerd to every extent possible, I am constantly trying to figure out how to use technology to make things better, easier, more efficient, less complicated, and able to fit into my life, while providing a product or service that has some cool factor to it. Every once in a while a few products come along to do that. I want to talk about some of the capabilities, but first, here's what spurred this:

"Restaurants will need to wholeheartedly embrace smart dining or risk their brands quickly becoming stale."
- Hospitality Technology Magazine, Sept. 17 2014. (Sorry, couldn't find the author)  

Smart Dining. That's definitely a way to look at it. From Yelp reviews that encourage users to game the system, to scrolling through a bourbon selection last week at CUT, technology is not only feeding restaurants in the most creative ways possible, but it's allowing a level of relevant interaction that is unprecedented, in a space that celebrates personalization. We're on the cusp of some amazing things, my favorite being those that require very little human interaction to fuel, but provide an atmosphere that is truly unique. This all hits pretty close to home for me and the team at Scrape with what we've been working on, but its one of those things that I never get tired of dreaming about. (The next trick up our sleeves is using location awareness to change the color on Philips HUE bulbs to highlight empty seats in a restaurant so that people can get seated and start spending .. ahem, eating, quickly.)

ApplePay coming along with the OpenTable integration is definitely going to change things in the space, but if you haven't heard of or seen TabbedOut in action, there is so much more that can be done that just paying your ticket. Granted, in Texas we're pretty proud of our locals, but this is one of those services that has the capability to increase efficiency, provide better customer service, and put me in more control of my dining (or bar hopping) experience everywhere.

So now that the problem of paying tabs and booking reservations is easy, what about when someone shows up at the restaurant or a busy streak is hit? One of my personal faves that I have seen is NoWait. It seems pretty simple to get in the hands of the restaurant, and it gives them incredible amounts of data.

What I see here though, is a huge open opportunity. Using things like social profiles, why not take this a step further for customization? Why not already know who is coming into the restaurant before they come in the front door, assuming they're a return visitor to the location or chain? By combining different technologies, I see that there is a huge opportunity. The thought that you could know who's coming in, what their food preferences are, what they are celebrating, what they might be interested in eating based on past history, where they like to sit, what the day for a living, etc, to me, is amazing.

I guess I don't see the future of the industry as merely a Smart Dining experience, I see it as an opportunity to do more than just provide a great restaurant visit. By leveraging data, location, transactional simplicity, customized visits, etc., I see this as a way to create a unique experience for each venue that allows technology to assist. From having a crowd-sourced playlist to being able to walk out with an automated tab, the times they are a changin'.

Let me put this out there: what I would love to see is data driven customization and personalization in the restaurant industry. I am not talking about a dining room full of robots. There HAS to be human interaction, I'm not at all saying replace the staff. I'm saying give them data and tools to create a better experience. I want, I crave, being able to have a great recommendation on appetizers, beer, wine, or a main course. I can't even imagine how great it would be to know that every time I sit down something is presented to me that will give me a great dining experience. I want the best of the restaurant all the time, any time. With technology, we can do that..and naturally, I'll be more than happy to help. :)

Friday, September 12, 2014

No Luv for Southwest Wi-Fi

To say that I became a little obsessed with my terrible Wi-Fi experience on Southwest Airlines from LAS to HOU today is an understatement.

As anyone who may have been a follower of mine for the last 4 hours and has since decided to mute me will tell you, I was a little upset. So, instead of just randomly venting on Twitter about the whole thing, I figured I would be pointed and a bit constructive.

First, let me say how much I appreciate Southwest Airlines Customer Service. A kind fellow name Sean took care of me via Twitter and immediately had the $8 charge reversed, while still allowing me to use the painfully slow access to vent and investigate. So, kudos to you guys Southwest for not kicking me off even though I was hating for 2 hours on your service. Well played, and much appreciated. Despite what I said, your customer experience will keep me coming back as a customer, as long as I don't have work to do on the plane. Fair?

So, what's the issue?
Check this out: one of the great things about SWA is that they let you watch all the video you can ingest from a few channels through a partnership with Dish. Cool huh? Yeah, if that's all you wanna do.

Without actually having any under-the-hood privileges, I'm gonna make some assumptions:
1. The video is pushed to a local server on the plane and then fed to the customers. Multicast style, right?
2. Users are so swept away with the video offering, watching the game, TLC, and what not, that they do that the majority of the time .. unless they're a nut like me who's trying to do a few things at a time.
3. All of that video is basically on-net at that point so the quality is perfect. And it is perfect, beautiful streams to tablets, mobiles, and laptops alike.

Here's where I *think* the issue is: the Access Point on the plane.
Here's why:

I dug around as much as I could to find out who on God's green Earth was responsible for the Southwest Wi-Fi offering. I even reached out to @FlyerTalk on Twitter to see if they had any thing I could reference.

You bet they did, 5 pages worth of posts dedicated to how terrible the Wi-Fi is on SWA. Some even hitting directly on the head of what I thought part of the issue was:

What I found was that a bunch of years back Southwest awarded the contract to a group called Row44. Row44 was acquired by Advanced Inflight Alliance back in 2012. Since then an award was made to Kontron to take care of the in-flight entertainment and inflight wi-fi component of the system.
So, Kontron has the contract via Row44, and they deploy this thing:

Even if you have very limited experience with Wi-Fi you'll probably notice the batwings on this thing. A trained eye shows you it's a Motorola 7131. But hey, it lists that on the product page, so if you don't know (as Biggie Smalls would say) Now ya know.. 

So, all that being said, let's look at what we have. A satellite feed to the internet from the airplane, a media server pulling in feeds from 8 or so networks, a Motorola 7131 AP distributing the service, a bunch of clients connecting. On my flight, I used my trusty super fancy AngryIP tool to show me 16 users pingable.

If, let's assume, 80% of those 16 are streaming video, let's say 13 users, at a pretty high bitrate from the media server, lets say 4mbps, you get 52mbps steady flowing through that device. On the network connection side, if it's pulling in 8 feeds at 8mbps each, that 64mbps on the satellite link. 
I don't know what the capacity of the sat feed from Row44 is, but that's gotta be taxing it .. jus a lil bit.

UPDATE: it's 11Mbps. Ouch. Here's what I read:

On the AP side, pushing a steady 52 mbps across that access point to 13 clients might be making things more than uncomfortable.

Let's reference Keith Parsons fantastic Wi-Fi Stress Test Report from last year to see some digits.

Enterprise class AP's under a load similar to that of what we are dealing with basically begin to implode at around 15 user mark when streaming video. Even more fun happens when you look at error rates. 

Look at that.. right around the 12-15 device mark.

Truth be told, I had forgotten what the specifc numbers were from this report until I just now opened it to grab some screen shots. The fact that it's right on the money isn't a surprise, it's just reassuring that Keith and I both know what we're doing :)

So, how do you fix it?
Well, Southwest Airlines, all ya gotta do is call. :) Your APs are wetting the bed and making customers like me think twice about who I fly, all because of your WiFi. If you don't think its a big deal, all you gotta do is wait. Wait for the iPhone 6 to start sucking down even larger amounts of data. Wait for more carriers to adopt T-Mobile's wi-fi texting and multimedia messaging. Wait for more customers to move all their apps to the cloud and demand higher amounts of service because their Dropbox is updating and their Office 365 won't load. 

I hope this helps ya. And if you need anything, my contact info is on the right of your screen. I can help you fix this.

Monday, September 1, 2014

Making Your Private Network Public. Hello, World. Got DHCP?

I want to talk about one little issue that keeps popping up over and over to reiterate something that might be overlooked, as it is so many times in the field:

DHCP Scope Size on Public Wireless Networks

It sounds ridiculous, I know, but I've had to troubleshoot so many networks where the DHCP scope was not configured to:
  1. Accommodate the number of users that were anticipated to join the network, 
  2. Have the appropriate lease time set to expire out the clients that were no longer there.

My advice is to keep in mind the combination of those two items in order to properly plan out your network. Building out a public wireless network isn't just about providing great RF coverage and signal quality, it's about making sure the network capacity is there, and the network is set to support the onboarding of so many new users.

As a best practice what I do, for a number of different reasons including security and networking configuration, is set the access point up as a DHCP server, if it's supported, for the guest network. Grant most of the deployments are single AP in retail, however in an enterprise environment there is usually a controller or cloud service that allows me to do to support multiple APs. Since the guest network is usually the lowest priority and offered as an amenity, I don't have to worry about routing or static IPs, specific ports being opened up, etc. Segmenting it onto it's own subnet and enabling client isolation on the AP is usually the recipe I use.

One of the key reasons that I started to use DHCP services on my own equipment, aside from any networking configuration the end user might have, was for ease of install and desire not to tinker with something on the customers existing network.  

I don't have to tell any integrator how many times I've been to the customer site, plugged in a piece of equipment that does nothing other than receive a DHCP lease from their server, and I'm the one who ends up destroying the network because of a static IP issue. The way I figure, the less interaction I have with their network, the better. Most retail plays and a majority of one-offs are an over-the-top service to their existing infrastructure, so this saves me and the customer some heartache.

The one thing that I think gets overlooked the most is the DHCP lease time. If you have a router that's unable to set the DHCP lease time and you're thinking about making your network publicly available or offering free Wi-Fi, go buy something that supports it. 

Secondly, when setting your least time think about how much time the average user spends at your facility, shopping mall, coffeehouse, etc. and set the lease time according to that. Each situation is different and there is no specific time that fits for everyone. An hour at a coffee shop, 3 hours at a clothing store, 6 hours at a mall, 8 hours at my pediatrician (seriously, she must have the worst waiting room time EVER.) The key here is to have the leases expire as soon as they can so that everyone has a fair chance at getting on the network. Keep in mind, the lease will be re-requested at the halfway mark; if it's set for 3 hours, 1.5 hours into it a DHCP transaction will fire off.

I've seen it happen too many times where the size of the pool is usually appropriate but the least time is not appropriate (usually it's set to the factory-default 24 or 72 hours). That means you have to wait a day or so before those leases expire. For the next 12 to 36 hours, those clients that have long since gone are taking up spots that could be given to other users who are trying to use network resources.

In the almost perfect storm case of a local shopping center in the Silicon Valley area:
The least time was set to 72 hours, the standard default setting up at Draytek router. There's a great restaurant in there that usually has a great lunch and dinner crowd. All it takes is a busy lunch and dinner where before you know it 200 leases are dished out .. and hung on to for a day and a half. By the time the lunch crowd comes in the next day, they're down to only a few leases that have expired out, and by day 3, there are no more or very few IPs.

All in all, I guess the point of this whole post just to make sure that you check the DHCP server when you decide to offer free Wi-Fi access. There's so many things that can go wrong with the network, taking for granted the DHCP is working fine is a simple oversight they can affect everything. 

Friday, August 22, 2014

A New Entry in Retail Analytics brings DPI to Wi-Fi Big Data. Welcome Fortinet!

The retail analytics space just got another new player in the game: Fortinet announced yesterday availability of presence and analytics. Teamed up with Kiana Analytics out of San Francisco, Fortinet comes to the game a little later than some of its competitors, however the power of the FortiGate UTM device makes them a formidable contender out of the gate. To learn about their product via a webinar, sign-up here/

What intrigued me about this new system was the inclusion of deep packet inspection. We have already explored what can be done with location data, analytic data, and social data. now let's talk about deep packet inspection. One of the coolest things that is demo'd is the DPI of data streams from a customer store.
Here's the scenario:
A user jumps on the free Wi-Fi offered at the store so that they can showroom, or shop, the store prices. For example a Kitchen-Aid mixer. Let's say it $199 at Target, but the user uses the Wi-Fi to look it up on Amazon to see if the purchase price is cheaper .. it is: $179. The shopper decides to wait on the purchase and buy it online. As the shopper makes his way out of the store, a digital display pops-up a coupon saying "Hey man, I know you want that mixer, here's a coupon for $20 off if you buy it now".

How'd it happen? Why $20?
How'd it happen? The DPI component of the FortiNet device is watching and waiting for search queries from Google, Bing, Yahoo, etc. When it sees a search query come through, it strips the query down, grabs the product item and the online price, compares it to the in-store price, determines if it can cut a coupon for the difference, and the waits for the customer transaction. If the customer transaction doesn't take place (location tracking combined with dwell time at cash register location ... duhhh) then it issues the coupon on the display before the user exits the store. Or pops it up on the phone. Etc. Etc.
Mixer: sold.

As a FortiNet partner, I have come to rely on its capabilities and service with almost every installation that we do. About 90% of the customer installs I perform include some type of FortiNet component. They're great at mitigating threats, providing a secure working environment, and a centralized interface to control everything that goes into and out of the network. When you combine that with the capability to supply deep packet inspection and reporting information on retail presence and analytics, it seems like a match made in Heaven for an already powerful network device.

I think they really hit the nail on the head with this latest feature set, knowing that their customer installs are all over the place. Being able to provide this type of wireless insight on top of the existing network data is a goldmine. I just hope they open it up to allow other manufacturers access points to send data to the device / cloud service. Don't get me wrong, I dig Fortinet but would love to have a few other AP vendors be able to feed it (AirTight, Ruckus, Xirrus, Aruba, Meraki, etc). While I'm dreaming, I'd love to see support for other devices (sensors, iBeacon, etc. be able to send data) :)

The A/B comparison testing on Kiana is fantastic!

The analytics component that they are featuring is not a homegrown solution. FortiNet has partnered with Kiana Analytics out of depths of Silicon Valley. I had not heard about Kiana before this, so I was anxious to find out as much as I could, as quickly as I could. I had a great call with the CEO of Kiana that ran the rounds of retail intelligence, big data, visualization, actionable insights, and more. It was awesome to have that one on one time with Mr. Fathi, as I am sure his world is about to get turned sideways once the FortiNet marketing and sales teams get a hold of this and dig deep into their user database for sales.

Here are some key things that I liked about the demo that were clear standouts:

Loyalty and campaign management give you the ability to track marketing efforts with visitor performance.
Intuitive and easy to use graphs and visualizations are a key strong-point of the Kiana service.
Heat mapping of clients using RSSI with animations for traffic patterns.

In our conversation we talked about the important things that drive retail analytics and what that means to the end user. I'll share my thoughts:

1. Visualization
Big data is great, but without the proper graphical representation of it, its useless. Store owners, specifically retail and hospitality, seem to be the prime target with wireless retail analytics. If you present them with a CSV full of numbers and percentages, you might has well turn into Beaker from the Muppets and start talking in beeps and blips. Visualizing the data and having it directly reported with little or no interaction is paramount. Don't show me the thousands of numbers and decimal points, give it to me in a big picture that makes sense.

2. Monetization Through Actionable Items
You all have heard me use the example of understanding the customer trends and then acting out management tasks or environment changes to counteract those trends. That's the key. Don't just tell me that people aren't there from 7-8, tell me what how to get them there from 7-8. This is the hardest part of all of this. Early on in my adventure with this stuff we hired a marketing professional to help analyze the data and direct the customers on what to do with the data. Take those fancy charts and graphs and figure out how to use them to drive business. The data is great, but its the action that is taken because of the data that makes the money, and that's what pays the bills.

3. User Experience
All of the data combined can be a phenomenal way to deliver customized, personalized, and overall beautiful user experiences. Take advantage of that. When looking at a retail analytics solution, don't just look at it for the data it can provide the shop owner or retailer, look at it for what it can do for the customer through the store. Alter the music, change the ambient color of the LED lightning, customize the menus and displays, etc. With all of this data at your fingertips and the ability to build each user visit custom tailored, offer something that no one else offers!

One of the features that I really got a kick out of was the ability to search for "VIPs" by MAC address in the system. The results below show how much time a user has spent at certain locations, either at a single store or multiple stores, and what section that user spent the most time at.

All in all, I'm excited to see this entry onto the field. As a FortiNet partner its exciting to see them doing something fun, and as a retail analytics nerd, I'm even more intrigued. This space is picking up some serious momentum and I can't wait to see where it goes next. As was said by a friend, "I guess mobile tracking is table stakes now, time to kick it up." I for one can't wait!

Thursday, August 14, 2014

The McAllen Creative Incubator gets a new Wi-Fi network!

One of the greatest things about doing what I do in the local community is that I get to go visit places that I haven't been to in a pretty long time, or that I went to as a kid and get to see them from an entirely different perspective. One of those places is the building that used to be our old local public library. Throughout the day today I've been upgrading their wireless network to introduce brand-new capacity, capability, and connectivity, not only for the guests that visit the building, but for the tenants of it as well.

Where the old library once thrived is now the McAllen Creative Incubator, a facility that encourages creativity in the arts and hosts art studios & schools, music schools, a local LPFM station, and open space for people to come and be as creative as they wish.

There's always something fascinating going on with the incubator, but up until recently they've been working off of a hacked version of a Netgear wireless router running DD–WRT. With 18 different offices inside this building and about 30 or 40 people there, teaching classes to 20 to 100 children at any given time, I felt like it was a pretty important initiative to get some new wireless connectivity in that building.

When you're trying to teach children and teenagers how to be as creative as they possibly can, the last thing you want to do is limit their capability to do things like get online and access rich content on the web. One of the tenants at the incubator, an LPFM radio station called KCYP was started by a buddy of mine named Joe Martinez. A couple of years back he got the itch to teach children about broadcasting and radio.
Now that it's been on the air for a number of years, it's great to see that they've made an impact on so many children's lives by teaching them about how to do something that they cherish, which is listing to music.

Sidenote: The RF nerd side of me wishes that I could teach some classes about RF signal propagation and different types of modulation rates so that maybe I could encourage some of these kids to grow up to be RF engineer. :) Anyhow..

One of the things that was asked of me today when I was installing this network by Joe from KCYP, was the amount of capacity available so that he could start streaming live WebCams from his studio. They currently stream all of their audio over the Internet but now he wanted to be able to show the faces behind their broadcasts online, encouraging friends and family to tune in. I think this is a great opportunity to leverage the connectivity at the incubator, however it wouldn't be possible unless there's enough wireless connectivity to support these types of endeavors.

We installed Ruckus Wireless 802.11n and 802.11ac products around the facility to serve the capacity and user-loading situations. We used 802.11n in the common areas, and 802.11ac in the specific meeting areas where there will be lots of client-to-client communication. Eventually we'll probably upgrade it all to AC, but for now this keeps the cost down while providing reliable, speedy, and well balanced throughput.

One of the key reasons we pushed AC into the meeting areas is because we are moving our CODE#RGV events over to this facility. CODE#RGV is a social hacking project where anyone who's building something .. an app, a website, a database structure, something, anything .. is welcomed to come and look for help, donate their help, and contribute to larger group projects. We've outgrown our space at IMAS and look forward to bringing more nerds out of the woodwork, and the Incubator allows us the room, schedule, and central location for it. If you can imagine 30-50 geeks parked in front of laptops pushing and pulling a ton of data, that is CODE#RGV, and that's why we need that 802.11ac capacity :)

McAllen Chamber of Commerce has been kind enough to do business with us and allowed us to help change the lives of not only the tenants of the incubator, but of all the children that they reach on the daily basis. Combined with the network we built at IMAS earlier in the year, I'm glad that we were chosen to provide the solutions for this and future community projects.

All in all I take a lot of pride in my city and a lot of pride in the projects that we get to work on down here. From my perspective, its more than just hanging APs, its providing something to people who may not even know we did it, but they'll be ecstatic that its there. :)

Monday, August 11, 2014

Re: Liking One Social Wi-Fi Case Study- and Disliking Another

Lee Badman wrote a killer counter-point write-up to my preso at WFD 7. Naturally I had to respond :)
Here's the link to his posting: Liking One Social Wi-Fi Cases Study and Disliking Another

Check out Lee's comments at the roundtable here

Lee, what a great write-up! I think you nailed the counter-point perspective perfectly, and I am happy to read, and respond to, what your concerns are.

I'll start with this:
The information, in the case study that we are using, that is gathered from the profile is not stored for more than the user session in the environment. Why? Because of the "Christmas Music" factor. If I retain your info and hang onto it, what happens when you're listening to Christmas music in December and you return to the AP in July? I don't care about your previous musical taste because it's not relevant. Relevancy is the whole key to this thing. Sure, there is data that can be collected and used to draw conclusions (top 10 music lists, top 40 artists of the year, etc.) but that data doesn't have to be tied to any personal data whatsoever.
The key here, again, is to deliver relevancy to the end-user by using the most current and up to the minute information possible. Music, food, experience, etc. all play into that, but only for the life of the session. Transient data my friend .. definitely saves on "big data" storage costs as well :)

Second, the opt-out list.
A number of manufacturers, with a constantly growing list, support initiatives such as Smart Store Privacy ( This gives you the ability to black-list or opt-out your MAC address from not only Wi-Fi tracking, but low power Bluetooth as well for future iBeacon stuff.

In addition to both of these points is MAC hashing, or user hashing. Sean Blanton from AirTight went over this a bit; in short the data collected is linked to a hashed ID when collected. This keeps you, you, and turns your data into another "thing".

In our examples, to address your bullet points:

  • If I opt in, can I opt out? (The example here is a bar- what if I’m crocked when I opt in?)
  • If I opt out, can I ask that anything to do with me personally be deleted?
    Dependent upon the person who you granted access to and built the application, maybe! It's up to each individual developer.
  • Can I expect that anything to do with me that was sold to others in exchange for “free” Wi-Fi be deleted from those other data stores as well?Again, dependent upon the person that you originally shared the data with.
  • If it becomes common knowledge that my personal life preferences are manifesting through the establishment’s environmental reaction to my presence, how might a stalker or identify thief leverage that simply based on what they observe, even if they don’t know my name?Knowing your preferences in things like music or beer selection, while I guess could be part a bigger picture to stalk you, are probably some of the end-trails of the information a stalked would use to go after you. I mean, if I was stalking you, I'd dig through your trash first and stake-out your house .. just sayin :)
  • What if “the algorithm” somehow gets it wrong, and turns me into someone I’m not based on what it reads in my profiles and shares that with the outside world through interactions with me at the establishment?
  • What if the algorithm gets it wrong, and sells my flawed persona to other companies who now think I’m someone I’m not?Those two points are totally awesome. If you, like Sam mentioned at WFD7, game the data, there's no telling what will come up. However that's the outcome of trying to trick the system though, right? Garbage in, garbage out? :)

Like I said, overall I am stoked that you approached this, especially on your blog. It was a pleasure to hang out with you and I hope to keep this conversation going. This is definitely going to be one of those topics that is never answered satisfactorily to anyone's expectation, but it's great for conversation!

Community / Municipal Networks and Classroom Extension for Rural South Texas

An article got published in our local paper with a few lines I was quoted on regarding citywide wireless access.
"Misison to Set-up WI-Fi Hotspots for Students"
All in all I think it's a great idea to start to look at classroom extensions in more areas like ours. I am a firm believer in extending education into the community where possible, and I continue to work on projects like this anywhere I get the opportunity to. From the inception of these technology pushes to get tablets and Chromebooks into students hands, I have always asked "why not infrastructure first?"

Bridging the digital divide and providing access to rural parts of our country is a way to use tech that I absolutely cherish. Growing up in an undeserved area has given me a perspective into these types of communities. When someone can't afford to heat their home or buy clothes, how can they justify even $15 a month for internet access? I am a strong believer in helping out the entire community for the growth of the city or town, and I think this is why its always been a soft spot for me.
Through Point to Multipoint wireless instead of expensive fiber connectivity, the decreasing cost of outdoor access points, and now the backing of the FCC and government with the new E-Rate rules, pushing Wi-Fi into classrooms and communities will hopefully become more commonplace. If we can leverage assets from cities and municipalities to help usher in this new age of at-home connectivity for students, imagine the possibilities? Remote learning, increased access to educational resources, the ability to learn at a pace and program customized to the students, etc. I think these are just a few of the benefits we'll see in the upcoming years with these types of classroom extension

Just a few days before the polls closed, the re-election campaign for Mayor Salinas dropped this YouTube video, and it got a couple of us local tech guys excited.

In an environment like these two local districts, Sharyland ISD and Mission CISD, they have started a limited deployment of technology to the students, while looking at the bigger picture of constant connectivity. I think this partnership between City and School District it's a great approach.  The statistics in our area, 37% adoption rate (the lowest in the US), has always been something I have thought needed some significant improvement, especially when talking about community / classroom extension. No one entity can do it alone.

In early 2012 I got invited to join a friend, and local super nerd, on the rooftop of a building in the City of Mission to provide free Wi-Fi during the local Citrus Parade.

A great view of the main street, Conway Ave., in Mission, Texas.
Not the best desk, but it worked for the day.

It was a proof of concept to help bolster the efforts he was mounting to build a community based network, so I was happy to help. Using some inexpensive off-the-internet gear, the network quickly got bombarded with requests to join.

Shaine syncing up mesh units and testing antenna locations.

It was a totally awesome test and it showed that there was a very large interest in the Mission community for Free Wi-Fi. Since then, Shaine Mata (who's also quoted in the article) has joined our team here at Frontera to help build out projects like this, as well as lend his efforts to a number of other things we do. It's exciting to see it come *almost* full circle from that rooftop. Heck, at we have a newspaper article that is spawning some conversation, so that's a great start :)

I'm excited about what we are doing down here in South Texas. We have a few cities that are already involved in, or working on, setting up connectivity solutions for their residents. The City of Rio Grande City cut the cord on their downtown Wi-Fi hotspot to kick off Phase 1 of a bigger initiative earlier this year, a project we designed, engineered, and installed. It is already getting huge points from the local school district for providing students a solution for the lack of broadband at home!

Me (Drew), Rio Grande City Mayor Ruben Villarreal, Darel (Frontera)

I hope to see more communities realize the potential of these networks and keep involving more school districts to partner in making community efforts a priority for helping students ... and remember, feel free to call or email if you need help designing, engineering, integrating, or purchasing this type of solution :) </shameless plug>