Wireless Cowboys

It is now 5 years since Katrina hit New Orleans and changed the face of the Gulf Coast forever.   One of the good things that came out of this disaster was the outstanding effort by wireless ISPs that came together to provide Internet and phone services to thousands of refugees from the storm.    Mac Dearman stood at the center of that effort.

I called Mac the day after Katrina hit to check in on him and see how bad off he had it.   Other than a little damage, his network was in good shape.   I called a couple of days later, and he told me stories about the refugees of the storm, churches and makeshift shelters filled to overflowing with people that had nothing more than the clothes on the backs.   He and his employees had been working non-stop to put in Internet connections and voip phones at the shelters so that the people there would be able to contact their loved ones and start the process of applying for federal help.    I could tell from the tone in his voice that he was completely worn out, but could not stop because this work had to be done.

I got on a plane the next morning and headed down to help in any way that I could.

Within two days after I arrived, there were at least 30 people camped out at Mac’s farm near Rayville, Louisiana and semi loads of donated equipment had arrived that allowed us to put Internet, VOIP phones and computers at nearly every shelter in Mac’s service area.   I had to leave after a week, but Mac took his volunteer army of WISPs down to the Bay St. Louis and Gulfport areas along the coast and kept going until the next spring.

It was truly an amazing effort, done with no government support, purely with volunteer help and donated equipment.   The campaign to help people after Katrina was a pinnacle moment of the infant WISP industry, and a perfect illustration of the ability of WISPs to provide critical services quickly, efficiently and professionally.

Thank you Mac, and thanks to all of the volunteers that were able to take the time to help him out.   WISPs everywhere owe you a debt of gratitude.

More reading:

http://www.redherring.com/Home/15053

http://www.cnn.com/2005/TECH/10/03/mac.dearman/

(This was part of an email discussion on Gordon Cook’s Arch-Econ list, slightly modified for your blogging pleasure)

The US Broadband network is “good enough” to be satisfactory for nearly every practical purpose.   1 to 2meg speeds with decent latency is good enough for voip, telecommuting, vpn networks and 99% of the practical uses for a broadband connection.   It is not quite fast enough to do full screen, full motion videoconferencing and it is not quite practical enough to deliver streaming HD Video content, and that is about it.

Case in point – I have a 100meg fiber connection at my office.   It is great for videoconferencing and downloading large files, but 99.99% of the time, I don’t need that much bandwidth.   I can do pretty much anything I need to do on my 4meg home connection.   My five year old pickup gets me where I need to go – from a practical standpoint it has the same utility as a new pickup – or a Ferrari for that matter.   Not everyone needs a Ferrari to get to work.

More bandwidth is required for the content companies to continue to feed garbage to zombie consumers on the other side, that is the real motivation for higher speeds.   Looks like it has done a lot of good for South Korea, where an entire generation of kids are turning into gaming zombies on the other ends of their world class broadband connections.   http://www.guardian.co.uk/world/2010/jul/13/internet-addiction-south-korea.   Several years ago, I had an employee that I was pretty sure had some kind of a drug habit.   He was habitually late, seemed to be constantly overstressed and was always making mistakes in his work.   After watching his work habits degrade over a period of 45 days, I stopped by my NOC at 4am on a Sunday morning and found him camped out there playing Everquest.   He had been there since 7pm on Friday night, fueled by cigarettes, Mountain Dew and pizza.    I have several friends who have similar problems controlling their gaming habits, and I have had my own issues with it.

Secondly, far too few people in the fields of law, academia and policy have enough appreciation for the fact that networks are not free.   It costs money to build, maintain and expand networks and there has to be an appropriate return for the companies that maintain these networks, and that is not understood very well by people who plug their computer in and just get their connectivity without thinking about where it comes from.   Spend a week in my shoes and you might have a little different appreciation for the effort needed to deliver broadband.   I am just as disgusted as anyone else that AT&T, Verizon et.al are banking enormous profits on their networks when they should be putting that back in to reinvestment, but the real problem is that we don’t have competition that forces them to reinvest.

I am not against better broadband networks for the US, but I am against misplaced allocation of financial resources in the form of the stimulus program and USF subsidies to telcos that claim they are needed to improve our broadband standing in the world.    We need more spectrum and fewer restriction on what independent operators can or can’t do with their networks.   We also need patience – its going to take time to get fiber to every home and we should continue to expand access to “good enough” networks to the people who don’t have any alternatives.

My preference is to continue building toward the ultimate goal of broadband abundance, but to do it in a financially responsible way that encourages more competition and less government assistance.   I recognize that faster broadband has benefits, but those benefits are trivial compared to the problems of climate change, energy resources and food production.   Broadband doesn’t benefit anyone if that person can’t eat, doesn’t have electricity to run their computer or happens to be dead.  That is my reasoning as to why broadband access should not be so high on the list of priorities.

Today I am settling in at the Renaissance Hotel in St. Louis for the WISPA Regional Show. What a great event this is going to be. This is the first time in a long time that there has been a gathering of WISPs and WISP vendors of this size. It is great to see many of my old long time WISP friends here, including John Scrivner, Rick Harnish, Mac Dearman, Patrick Leary, Jack Unger, journalist Alex Goldman and a great group of vendors. We kick off tonight with a pre-registration party, then tomorrow we will have what should be an awesome presentation with the “Pioneers and Visionaries” of the Wireless Industry, a lunch keynote with Michael Calabrese from the New America Foundation and three tracks of WISP content. Thursday kicks off with a WISPA Board meeting – the first one where the board will meet in person instead of online – and then keynotes from Peter Stanforth of Spectrum Bridge and Julius Knapp, the FCC OET Chief. It’s going to be a great event!

The time has come to kill the Universal Service Fund, and make sure that it is not replaced with another Universal Broadband Fund that will do nothing but line the pockets of the same telcos that have failed to deliver.

An incumbent that can’t survive without USF subsidies SHOULD go out of business so that they can be replaced by an entity that is able to survive in the market.   Fast Failure is far better than dragging the decline out over decades, while a government supported collects the money and lets the infrastructure rot away.

If an ILEC could no longer get their $150/month/line subsidy to support the market and they can’t keep it supported, they would turn around and sell it as fast as they could get rid of it, to the point of practically giving it away.  The best result would be that inefficient, government supported telcos get out of the market and are replaced with entities that are better suited to take care of the market.   The new players may not necessarily be other large corporate entities – they might be community coops, mid-sized regional or sub-regional players or even – gasp – WISPs that already have infrastructure and staff in place to handle the technical requirements and would be able to benefit from the right of way and existing copper plant.

Related to this concept – this is a giant piece of motivation for WISPs to get their billing, infrastructure, technical expertise and legal counsel upgraded so that they are ready to take advantage of these opportunities when they come up.

One of the mailling lists that I follow (Gordon Cook’s Arch-Econ list) recently featured a discussion about resilient communities (RC) that brought focus to many items of interest to me.   Here a link to an excellent blog that regularly talks about Resilient Communities.   Global Guerrillas

Essentially, RC is about making communities self sufficient in several important ways that will allow them to prosper even if they are disconnected from the national/global infrastructure.    Here is a definition of Resilient Communities from John Robb:

This conceptual model creates a set of new services that allow the smallest viable subset of social systems, the community (however you define it), to enjoy the fruits of globalization without being completely vulnerable to its excesses. These services are configured to provide the ability to survive an extended disconnection from the global grid in the following areas (an incomplete list):
• Energy.
• Food.
• Security (both active and passive).
• Communications.
• Transportation.
The resilient community has broad applicability beyond just improving the ability of those of us in developed economies to preserve wealth and a quality of life despite severe system shocks. It can also be applied to the problems of counter-insurgency in semi-modern urban environment (to radically update a process that was built for the last century) and provide the potential for organic development in underdeveloped areas of the world. The key is that we need to support the open source efforts currently underway to expand this capability underway such as the transition towns movement to MIT’s low tech solutions effort.

The concepts that have special interest to me are communications, energy production and food production.

WISPs, some rural ILECs and a few CLECs are the only LOCAL IP-based communications providers left in the US.    Outside of this sliver of the total telecom market, we have given up the control of the majority of our spectrum resources, lost local jobs to remote call centers and transferred larger portions of our income to national/international companies that contribute little or nothing to our  communities.     WISPs have an important role to play in the Resilient Community, as one of the few alternatives for locally controlled and maintained communications networks.

Energy production is another item of huge importance, as energy costs continue to skyrocket and there are little or no alternatives in place for local production.    Instead of taking individual houses “off the grid” – how can we take entire communities “off the grid” and regain control of our energy?   Developing alternative sources for energy outside of the traditional networks is going to be critical for independence and maintenance of all kinds of local facilities.

Finally, food production is a big point of failure for most communities.    What would happen if WalMart failed or major problems with the transportation network caused food prices to skyrocket?    In my area, I think that we could grow enough food locally to support the population, but the bigger cities not surrounded by arable land would be in big trouble.

I look at the idea of Resilient Communities as an insurance policy and an investment.   Building up local resources and supporting those resources reduces the amount of money flowing out of communities and improves our chances of maintaining a decent quality of life if/when any of the global networks that we are so dependent on suffers a major malfunction.

There is no dispute from me that fiber is the end all, be all of communication technologies. Love to have some to my house, but it isn’t happening soon for me or many of my rural comrades, so wireless plays an important part.

There is one important thing to keep in mind about wireless. There is a quantum leap of capacity difference between MOBILE wireless and FIXED wireless, and too many fail to make the distinction.

Mobile wireless makes substantial sacrifices in speed and performance due to the imbalanced and highly variable return path from a small antenna. There will be constant capacity issues, network overloads and complaints about service coverage on mobile data platforms forever. It has been massively overpromised and 4g/WIMAX/LTE will never be able to deliver true, usable broadband to mobile devices outside of a lab environment or an underpopulated area with few users and a strong cell site nearby. The exponential increase in demand for data and laws of physics will make it impossible to keep up with the consumer demands and expectations. Every provider’s mobile data network will suck, simply because multimedia streaming and the demand for mobile data will massively overload them.

Fixed wireless sacrifices mobility in exchange for a strong return path and a higher signal to noise ratio, which allows for higher data modulation rates and increased capacity. It is also far cheaper to deploy than mobile, highly scalable and not dependent on ridiculously expensive licensed spectrum (although fixed wireless in licensed spectrum can do some pretty amazing things). The combination of ubiquitious unlicensed fixed wireless equipment and the lower hurdles to market entry has also created a petri dish of innovation in services, business models and non-government subsidized expansion of broadband to unserved and underserved areas. Wireless middle mile is also far underrated, as 100meg/300meg and even gigabit wireless backhauls are now available for relatively sane prices and do a wonderful job of completely bypassing all of the tariffs, artifically created intraLATA crossings, facility construction delays and contractual handcuffs. There are many of the old AT&T Long Lines towers in my service area, and we have backhaul facility on several of them. They may not carry as much traffic as fiber, but they carry a lot more traffic than T1s and copper, and that is the true alternative in many of these places for the near future. Wireless middle mile doesn’t have to carry the same amount of traffic that fiber does, it just has to carry ENOUGH traffic to be better than what is already in place.

The innovation in mobile wireless is focused on the billing systems and marketing campaigns. This weird phenomenon of zombies who are falling all over themselves to get the latest piece of icandy, pay out the nose on a two year contract for AT&T’s awful phone/data service and only get access to apps approved by “Big Brother Jobs” kind of blows my mind. AT&T have created this sense of mass delusion that bypasses the typical person’s common sense and turns them into i-d-10-ts. Do not confuse this mess with the fixed wireless industry. One delivers usable broadband. The other delivers empty promises.

Here is note we received from one of our Medicine Bow customers that I had to share:

Message:I really want to thank you for coming to Medicine Bow, WY.
Early this morning I was ablt to have a webcamera call from my son, a US Marine on deployment. It was wonderful to be able to see him and know that he is OK. This is his fourth deployment and this is the first time we have been able to use the webcamera for a call from him. Without you this would not have been possible. A million thanks for just one smile from so far away.

What gets lost in these discussions of “how far behind the U.S. is” with regard to broadband penetration, is how much is really needed to be useful, rather than what the market observers, academics and even the customers think.

From my experience in the field, as both a provider and a user, I think that a good 2meg connection with decent latency and jitter is plenty for 95% of the applications that consumers or telecommuters are going to use.   The other 5% of applications are mostly video related (high definition video and videoconferencing).   Two years ago, I would have said 768k would be appropriate, but YouTube and Netflix changed the game considerably.

I work from home a lot, and I have a 4meg wireless connection with a 60gig cap.   Last month I used 9.5gig.   I use voip daily and video Skype occasionally, my wife downloaded her Adobe suite software last week and syncs up her home machine with her work desktop via Dropbox, my son watches his Lego videos and we grabbed Monty Python and the Holy Grail (among other movies) because he was fascinated with a YouTube clip of the Knights Who Say NIiii.

Everyone has different perspectives on this:

1)  Marketers want to tout the fastest speeds possible, and focus on more more more

2)  Academics look at theoretical possibilities of networking, and tend to jump past the immediate hurdles toward where they think we should be without giving the transitionary stages much consideration

3)  Customers almost always think they need more speed – except for people coming from dialup or satellite.

4)  Big ISPs and cellcos are trying to squeeze every last dollar out of each bit going through the network, and are in a conspiracy with the Marketers at #1 to promote the fastest speeds but with all kinds of bells and whistles attached.   EBIABE – Each Bit Is A Billing Event

5)  Smaller ISPs are scrambling to find the balance between all of these considerations and slowly evolve from where we are now to where we need to be.  It is a daily struggle.

What other perspectives do you see?

Bandwidth caps have been a trending topic of discussion lately, I thought that I would share the bandwidth cap policies that we recently implemented at Vistabeam, along with some details on how we are enforcing it and how we established the caps.

Going back to day 1, we have had a 3gig cap on broadband customers with a $25/gig surcharge for anyone exceeding that amount.    Fast forward six years, and that cap was so low as to be a joke – and we had not been enforcing it.   It was also very difficult to collect accurate accounting data since we utilize access points from multiple vendors and did not want the overhead or additional configuration hassle of PPPoE or PPtP for all of our customers.

After looking at several different options for collecting the bandwidth traffic information, we decided to use open source tools to develop our own solution.   We installed a switch between our core and edge routers and mirrored a port to our new collection server.    Daily reports are mailed out to our techs list to show the customer who are nearing or over their caps.   A customer page was created that shows the customers how much bandwidth they have used, how much they have left before charges and what their overage charges are (if any).    The customer page also shows their historical usage trend for the last 12 months – starting with April 2010 when we started collecting the information.   Starting on June 1, we will bill overages as a separate charge to the customers on the 1^st of the month, regardless of their billing anniversary.

The process of implementing this was quite interesting.   Out of 2000+ customers, 80 used more than 10 gigs for the month.   One customer – a 1 meg subscriber at the far eastern edge of our network, behind seven wireless hops and on an 802.11b AP – downloaded 140gig.    Another one, on the far western side of our network, downloaded 110gig.   We called them and found out that they were watching a ton of online video.    We discovered a county government connection that was around 100gig – mostly because someone in the sheriff’s department was pounding for BitTorrent files from 1am to 7am in the morning, and sometimes crashing their firewall machine because of the traffic.     We also discovered that there was 80-100meg each day of stateless udp type traffic traversing our routed network and getting to our core router.    Revised firewall rules on the APs fixed this problem.   The majority of the rest of the subs on the list were either online video watchers, people with virus problems or non-power users who had mistakenly left filesharing programs running on their computers.

After reviewing the usage records, we decided on the following cap sizes for our plans:

Package                                                               Monthly Download Cap

384k                                                                       10 gigabytes

640k                                                                       10 gigabytes

1 meg                                                                    20 gigabytes

2 meg                                                                    40 gigabytes

3 meg                                                                    50 gigabytes

4 meg                                                                    60 gigabytes

8 meg                                                                    80 gigabytes

Additional capacity over cap                        $1 per gigabyte over the cap

I feel that these caps are more than generous, and should have a minimal effect on the majority of our customers.   With our backbone consumption per customer increasing, implementing caps of some kind became a necessity.    I am not looking at the caps as a new “profit center” – they are a deterrent as much as anything.    It will provide an incentive for customers to upgrade to a faster plan with a higher cap, or take their download habits to a competitor and chew up someone else’s bandwidth.

This has been an educational experience, and probably one that we should have gone through a couple of years ago.

Apparently my tirade about broadband mapping reached a few ears in Washington, as the NE PSC called me this afternoon to let me know that the NTIA is willing to accept shape files and is willing to relax some of the data requirements in order to get fuller representation from WISPs.    Making ourselves heard and showing a willingness to be part of the solution is the first step to getting better results.
Here is a copy of the email that I sent to the Nebraska PSC today with my followup comments.   

—

I am writing with further comments and concerns about the Nebraska Broadband Mapping Initiative.  After participating in the conference call about the mapping program yesterday, I was left with several concerns.

My first concern is about the accuracy of the data that will be collected.    The number of providers that have not responded to the NDA request and/or the data request is very high, and that means that there will be substantial inaccuracies in the final dataset that will make the final results of the project flawed.   A dataset that only includes 20-50% of the total data needed could lead to policy decisions that could have an adverse affect on the smaller providers that cover otherwise unserved areas by encouraging government supported overbuilds.   This would be wasteful of taxpayer money and could put many of the smaller providers out of business, causing a net loss of jobs and the loss of broadband service to customers of those smaller providers.   It is critical that most if not all of the broadband providers in the state be represented in this project.   The attitude that the state contractor appears to have is that non respondents will simply not be included.   I would hope that this attitude will change to be more inclusive of the smaller, non-wireline providers who do not have the ability to generate the requested data easily.

My second concern is about the data that is being requested.   The data request template is asking for a lot of data that I don’t feel comfortable divulging to any outside entities, including customer addresses, GPS coordinates and frequencies used on our towers and the anchor institutions that we serve.   Many of the other WISPs that I work with are also not comfortable turning this information over to an outside party, even with the NDA.   After several discussions with other experts in the mapping and data collection field, I have come to the conclusion that the mapping requirements would be effectively served by delivering the GIS shape files of our coverage areas along with a summary of subscribers in each census block.   I have already delivered the requested shape files showing our coverage, and am working toward the census block summaries.   If the data requirements could be adjusted so that this information would be suitable, I believe that you would get more response from the smaller providers.

My third concern is about the cost for smaller, non-wireline providers to collect the data.  While most wireline providers already have shape files and geocoding information already collected and available, many wireless providers do not have this information readily available and do not have the tools or technical knowledge to get this information collected within the requested time frame.   Committing man hours to do this in-house or bring in outside assistance places an undue financial burden on providers that are often self-funded and would prefer to invest that money into their networks.   The grant was given to the PSC, not the providers, and yet we are being asked to spend our time and money to get this information together.   Coming up with a way to help provide the manpower and financial assistance necessary to collect this information would provide a win-win situation for the providers and the PSC and increase the amount of data collected.

Finally, I believe that more effective outreach could be established with the providers so that the comfort level is higher.    Sending an email with a large data request and a short deadline for response is not going to be received well.   A series of emails with detailed explanations of the program’s purposes and benefits to providers, an intelligently designed website with progress reports and followup phone calls to the providers who have not returned the information would go over much better.  WISPs have not been required to collect this information up to this point and there is no mandate for its collection, so it makes sense to build up a positive relationship rather than dictate what should be provided.   One benefit of this process is that it is an opportunity for the Public Service Commission to build a rapport with the WISPs and gain a better understanding of their place in the broadband infrastructure while educating them about the purposes and benefits of the Public Service Commission.

Thank you for taking the time to work through this process.   I hope that my comments reflect my desire to improve the process and get us closer to the desired outcome.

Best Wishes,

Matt Larsen

Vistabeam.com

wispdirectory.com