365 days after my business was destroyed by Hurricane Maria

Exactly today...

365 days have passed since Hurricane Maria made landfall on Puerto Rico.  For most people, it is a date they dread to remember.  Sadness and suffering are feelings that gloom over this fateful day.  

It's very difficult to describe in words the effects of a category 5 hurricane on your home, your family and your business. The last time a big hurricane hit Puerto Rico was back in 1989 when Hugo struck us as a category 3.  Since then you might think that we learned from that lesson and were ready for whatever fate had for us. 

Well, as most of you know, we weren't ready at all.  The island had been suffering for a 10+ year economic recession and the effects of it could be seen before the hurricane. After Maria hit us, almost nothing was left. Including my business. 

Aeronet...
was a project I started 17 years ago as a solution to a problem I had.  Back then I had no broadband Internet at my home so I decided to brush up my tech skills and my telecom background and after some research, I hacked together a wireless system to beam Internet from the retail business I had back then in San Juan 8 miles down to the suburbs on a small hill where I could then repeat it to my neighborhood. 

AeroNet's first website circa 2002

What started as a personal quest, soon converted into a 1-man shop/company, then 16 years later AeroNet grew to a 50+ employees enterprise that provides Business Class Telecommunications solutions to over 5000 customers in Puerto Rico, USVI and South Florida. 

All changed 365 days ago...

As soon as I could, I drove to AeroNet headquarters (about a 15 minutes drive).  I took me about an hour and half to maneuver over all the debris in the highway.  I was received by a small group of staff members that volunteer to stay at our offices, their faces told me all I needed to know.  Our whole operation was damaged by the hurricane, all our fiber backbone connections to the Internet were down, power was out throughout all our sites in the island, and we were by most standards out of business. 

Collapsed Tower

No time to quit... 

I had a huge responsibility on my shoulders towards our employees and towards our customers.  There were never any thoughts on quitting or balking at the task at hand. It was not an option!  We had to rebuild and rebuild fast! On Friday, September 22 we started rebuilding. 

The rebuilding process was an enormous effort that required the uppermost cooperation from all the team members at our organization.  During the weeks after the hurricane, we went to great lengths to accommodate all the needs of our employees and customers.  We provided shelter to some employees and their families, provided food, provided gasoline among other needs, but what really gave me most satisfaction was that we provided a home for everyone. 

We worked 7 days a week, 12-16 hours per day and we made it!  By end of November, our telecom network was about 80% up.  By the end of January, 98% up!  This great feat was really achieved by our great team.  All the members of our company contributed to this expedite recovery and have made us a stronger and more cohesive organization. 

Looking Back...

I learned from this experience what commitment, dedication, devotion, and, responsibility truly mean.  I learned them from my family, from our customers, from our team. I hope we never have to go through this experience again, but if we must, We Are Ready... We Are #PuertoRicoStrong

Follow me on social media for more information on this and other related topics: 

Gino Villarini: Twitter, Instagram

Hurricane Maria - Lessons Learned - a panel discussion at the CIO & IT Leadership Conference #PRisIT

September 20th, 2017 will be a date that will be remembered for a long time by many. To some, it was a day of tragedy, for me it was a date for maturity.  Maturity, because Hurricane Maria was the "test of fire" for our company.  The worst that could happen, happened and it depended on everyone as a team to overcome it. And we did! 

 

If anything Hurricane Maria taught us was that our "infatuation" with fiber optics connectivity needed some revision.  Many don't know the fact that wireless technology saved the day after the hurricane.  Even if many did not noticed!  Several companies like ours provided wireless Internet and data services to many critical operations including the Disaster Management Center at Centro de Convenciones, critical connectivity for Evertec / ATH, Hacienda, FEMA, US Army Corp. of Engineers, among many even including connecting cellular towers for mobile service. 

As of May 10th 2018 Sin Comillas, a local business news outlet, reported that the local cable provider customer base was 80% up.  Not to kick a dead horse, but this puts forward the fragility of wireline networks.  Our customer base was 80% up just 8 weeks after the hurricane.  I use this example to highlight the true advantage of a wireless based infrastructure.  Although is not Hurricane proof, it can be reestablished far quicker than wireline.

At the 2018 Puerto Rico IT Cluster: CIO & ITLeadership Conference, I was able to participate in a panel where the effects of hurricane Maria on the telecom infrastructure on Puerto Rico were discussed. Lessons learned and next steps to be taken were among the topics of interest.

 (1) Tell us about the biggest challenge you faced after Hurricane Maria

There were many challenges faced after Hurricane Maria, starting from the most important part of any business, Our Team.  A small group of our dedicated staff, stayed at our facilities during the hurricane, they were responsible for monitoring and logging all incidents related to the hurricane.  I thank them for their efforts, as they were the "first responders" of our plan. Like many others, just getting our staff with the basic necessities so that they could be able to work was a challenge. Food, Gas and even a place to stay were provided to our employees. Some of our staff members turned our facilities into their temporary homes, as many brought the rest of their family to stay during the day. Another challenge was logistics, including physical access to our towers, our customers and other locations for repair.  Also, related to logistics, getting supplies from the mainland turned to be a huge problem.  The federal government was controlling all inbound cargo to PR. 

(2) Can you share with us a brief summary of how you overcame this challenge?

We relied on our customers as partners to overcame many of the challenges.  Starting from reaching cooperative agreements with gas stations so that we could have gas to operate and for our employees. Also, we partnered with a restaurant that provided our staff and family with daily breakfast and lunch. 

The logistics challenge was one that was a bit difficult to overcome.  Thankfully Most of the principal roads were somewhat cleared for traffic by the local citizens, letting a hand to the local government to quickly recover access to transportation which is critical on this kind of event. 

Getting the supplies from the Mainland to PR in order for our company to quickly repair the damages presented the biggest hurdle as the US government was controlling all sea and air traffic. The most cost effective way of shipping 20+ pallets of equipment would be by sea, but we had to rely on more costly air transport to get our supplies in a timely manner. This with some issues in terms of being able to get an air transport window by the FEMA controlled airport.

(3) Where is your company now vs. pre Maria and how are you preparing to become more resilient

We are bigger and stronger! We have learned many lessons along the way and rectified many weaknesses.  Among our plans that are being executed right now are:

-Non-dependence on any government help, during a disaster, your company must count with all the resources necessary to operate completely disengaged from any help or assistance from the government.  This includes shelter, food, water, gas, transportation and communications. 

-Alternative long run power.  We have revised our backup power plan and expanded it capabilities from isolated operation to 4 continuous weeks.  The addition of solar energy is critical, but must be integrated property to have a high survivability rate. 

-Redundancy, Redundancy, Redundancy.  Any operation that relies on Internet access for its continuity, must include several, fully independent connectivity providers.  We all now by now that a fiber cut or a copper line can take months to get repaired so to guarantee the survivability of your business, other technologies like wireless or satellite must be included in the contingency plan.  We relied on fiber for backhaul but we are building a multi-gig microwave backbone that goes around the island and thought it. This network will protect our services for future events. 

Enrique Ortiz - President, Claro PR / Gino Villarini - CEO & Founder, AeroNet / Henry Gordillo - President, Intech / Felix Lugo - Country Manager, C&W Business.

Follow me on social media for more information on this and other related topics: 

Gino Villarini: Twitter, Instagram

NO POWER = NO INTERNET? Why your Business loses Internet connectivity during a power outage?

Has your business experienced an Internet outage during a power blackout? How can this happen even if you have a backup generator?

This problem is one that has been very common here in Puerto Rico, our market.  Puerto Rico was struck by two hurricanes in the past 6 months, most of the power grid was lost and it's still being reconstructed as of today. Power outages have become frequently common after the hurricane, because the local power grid is still very fragile. 

But the problem of losing Internet or Data connectivity during a power outage is one that is very common through all markets... even when customers have backup generators.

Why? 

All service providers rely on electricity to operate their networks; Fiber, Wireless, DSL, Cable all require power to distribute their services thought their networks.  Routers, Switches, Fiber OLT's, CMTS, Microwave, LTE Base Stations all depend on power to operate.  The way a network is distributed through a geographic area and how power backup systems are built for them contributes greatly on how a network withstands a power outage.  Let's take quick look at some cases:  

Fiber Networks: 

Fiber Optical Line Terminal (OLT) in a Data Room

Most fiber networks use two prevalent technology deployments, Active Fiber and GPON. Both technologies require Fiber Switches and/or Optical Line Terminals (OLT) to be distributed through the area they provide service.  Most providers use data-centers or data-rooms within their coverage area to establish POP's or Points of Presence where most of the customer fiber connections are grouped and connected to the backbone.  Is in this POP's that power backup systems become critical.  Fiber data equipment consume a reasonable amount of power, although most of the POPS have a battery backup system, they last for an average of 2-6 hours. A generator is a must on these locations. 

Cable TV and DSL/Copper based data networks:

DOCSIS Cabinet on a Pole

Both of these types of network are very similar on their layout and power needs. Cable TV based Internet uses a technology called DOCSIS and it requires that the data distribution equipment to be very close to the customer location.  A DOCSIS node usually serves an average of 500 customers and since it requires power, it is prone to power outages.  DOCSIS nodes are usually installed in street or pole cabinets, with limited space for long term backup options and/or  generators.  They usually last between 2-3 hours before shutting down.   

FTTN Cabinet - notice the limited space for batteries.

DSL/Copper networks operate similarly as in order to provide high speed internet and data services, most providers have moved in to a fiber to the node network (FTTN) layout.  FTTN means that a DSL provider puts a cabinet near its customers to be very close to them and provide higher speeds (distance affects directly the capacity and speeds of DSL network).  So very similar to the DOCSIS network, a FTTN network has cabinets within 2000 feet of its customers, serving between 500 and 1000 customer per cabinet.  They suffer from the same limitations as the DOCSIS networks:  almost no space for battery backups and no generators.  When power is gone, the FTTN network will soon follow. 

LTE Rooftop Cell Site 

Fixed Wireless / Microwave / LTE Networks

All three of these networks utilize wireless signals to reach their customers.  The equipment utilized between them varies by type of network or technology used but the concept is very similar.  Customers get their service via a radio enabled network that is usually located in rooftops or in towers.  These facilities can serve from hundreds to thousands customers, so less locations are needed when compared to a DOCSIS or FTTN network. On this locations or sites, the operators have an advantage because most of them have ample space to locate battery back up systems, generators and even solar power systems.   Here the service provider has more control on how long it wants it service to last during a power outage. It will mostly depend on how well a contingency plan is executed. 

Cell Site Generator

After hurricane Maria, some service providers have been working around the clock to ensure that their networks are resilient to power outages that are so common in Puerto Rico.

AeroNet Site with Solar Backup

Even before Hurricane Irma, AeroNet has been in the forefront of limiting its exposure to power outages.  By using extended battery systems, solar power systems and generators, we are able to withstand long power outages that affect your service. 

When choosing an Internet and Data service provider, take in consideration these facts, as they are directly associated with your business operation and continuity during a power outage event.

For more information on AeroNet, It's services, and how we could assist in your Business continuity and disaster recovery plans, call us at 787-273-4143 or please visit our website. 

Follow me on social media for more information on this and other related topics: 

Gino Villarini: Twitter, Instagram