Emergency FTTx Restoration Best Practices —
Natural disasters, hurricanes, ice storms, fires, earthquakes. Is your Outside Plant (OSP) at risk? Do you have a plan? You can design and build the highest quality OSP infrastructure, but Mother Nature is always there to test it. Are your infrastructure, technicians, and operations procedures, ready for the test?
If you are responsible for the operation of the OSP infrastructure, a natural disaster is always a concern. A hurricane, blizzard, winter ice storm, or flood, warning may allow only a few days to prepare. Will that be enough time for your system and crews? A fire or an earthquake could occur at any moment. In the event of an earthquake, distant and wide areas can be affected by ensuing tsunamis. In all extreme situations, catastrophic damage is expected.
This article offers guidelines to prepare your OSP infrastructure and crew to quickly respond to outages and damaged OSP infrastructure:
- Crucial points to consider before a disaster or outage.
- Minimize the effect of the event by implementing a routine maintenance schedule.
- Restore service faster and minimize costs with an emergency restoration plan.
Preparing Is Essential
Preparing for emergency restoration begins with the initial design and installation of the OSP infrastructure. The OSP should be designed with solid and proven engineering principles, and be detailed in its design. It needs to be built with premium materials designed for the environment, and to utilize trained, qualified construction crews.
Quality assurance inspections and on-site observations are a must to ensure that your teams are following the detailed design.
Documentation should be complete and include the following:
- As-builts and redlines.
- Duct routes.
- Maintenance hole locations (Butterfly drawings).
- Equipment, vault, and handhole, locations.
- Pole locations (all with GPS coordinates, if possible).
- Cable routes.
- Splicing records and testing results are needed for copper, fiber optic, and coaxial cable.
The records should be kept current and available with GPS coordinates, if applicable, to provide quick access to the availability of vacant ducts, spare fiber strands, and copper pairs if they are needed for restoration efforts.
Availability of redundant routing should also be considered at this time to help in either rerouting services or minimizing the effect.
Additionally, having a plan for spare equipment and ensuring the ability of crews to have 24-hour access to supplies, and additional labor forces (if required), is a crucial step in the ultimate restoration of service.
Maintenance Is Real!
Emergency restoration is triggered by an event, while maintenance is an ongoing process. At first, these 2 functions may seem unrelated, but a robust routine maintenance plan can greatly enhance disaster preparedness.
Once the initial design and installation have been completed, a maintenance plan needs to be implemented. At the core of any maintenance operation is the on-staff technicians performing the work. Technicians need to be readily available and have the knowledge, skills, and abilities, to perform the day-to-day tasks that are required to keep the OSP in prime condition.
The items listed below should be addressed during routine maintenance so they benefit the OSP infrastructure in the case of a natural disaster.
- Conduct regular pole inspections.
- Check that poles, messenger, guys, and anchors, are in good condition and up to code.
- Verify bonding and grounding.
- Ensure there are no unauthorized attachments that could create a dangerous or hazardous electrical condition.
- Inspect pole attachment zones, aerial terminations, and splice enclosures.
- Implement an annual tree-trimming maintenance plan.
- Confirm that all terminal cases, (aerial and buried), cabinets, and pedestals, are sound, labeled, clean, and with no wildlife intrusions.
- Check the conditions inside the pedestals, maintenance holes, vaults, and cabinets.
- Verify bonding and grounding.
- Ensure that vacant ducts have duct plugs in place.
- Replace or refresh pest control in cabinets, pedestals, and vaults.
- Travel infrastructure routes.
- Verify that route markers are in place.
- Confirm labeling on the poles, cabinets, and equipment.
- Perform Right-of-Way Inspections.
- Clear or trim trees or shrubbery, and confirm that the property owner has not blocked access.
- Inspect maintenance holes or tunnels to verify that they have not been damaged by traffic load, ground thaw/movement, or construction.
Given that maintenance staff often conducts an emergency restoration, ensure that all of your on-staff technicians have been properly trained on the products that they are repairing/replacing.
Another best practice is to establish redundancy in your crew’s skill sets, including cross-training and hands-on field restoration training.
A disaster area is not the time for new product introductions or training. The maintenance staff needs to be well trained in safety and standards to ensure that safe work practices are followed, and that the system is restored to proper standards and code compliance. An ongoing training program should train not only the initial skills for routine maintenance, but should also develop the workforce to perform disaster recovery or emergency service restoration.
Maintenance Plan Beneficial in Many Areas
In addition to ensuring a skilled workforce, a maintenance plan can be beneficial in many other areas.
- Complete documentation of the OSP infrastructure (and areas adjoining, if available) is critical
- If this is not already available, your technicians can gather the information during their day-to-day activities.
- Dispatch can process a work order to provide the crews with adequate time to gather and/or create this information on days where other maintenance requirements are relatively light.
- These activities would include documenting circuit layout records (CLRs for all circuits — copper and fiber), creating cable and pathway records with status (rod and mandrel history), maintenance hole (MH) butterfly detail drawings, and overall system drawings.
- Documentation of all currently installed equipment should be available for reference.
- As part of any maintenance plan, a well-documented storage location for repair materials and supplies is key.
- Technicians should be trained and familiar with the process for 24/7/365 access to materials, supplies, tools, and test equipment required for emergencies.
- They should also know how to charge, calibrate and maintain test equipment as well as reorder necessary materials.
Ideally, emergency restoration equipment, tools, and supplies should not be utilized for routine maintenance. Where this is not possible, critical test equipment, tools, and supplies that are drawn for routine maintenance must be documented to be immediately reassigned in the event of
The maintenance plan should also address the standard practice for what is considered routine maintenance and support of the system, and what is considered a natural disaster or emergency. A schedule for testing and calibrating all equipment (if required) should be included in the plan. This includes, but is not limited to, dielectric testing (rubber gloves, mats), meter calibration (OTDRs, LS/PMs, certification test sets) and annual maintenance records (bucket trucks, trenchers, plows), etc.
With a robust maintenance plan in place, the OSP infrastructure is in the best condition to withstand a natural disaster. A maintenance plan also provides a procedure to address most of the challenges faced during the disaster.
Once the restoration area has been deemed safe, the first action is to call a well-trained team that has the knowledge, skills, and ability to perform the repairs. They should also know the teams’ procedures and their specific responsibilities.
Be sure to remember that your technicians may have also been severely affected or impacted by the same disaster. They may be dealing with emotions that can impact their performance. Therefore, we recommend creating a technician-only escalation hotline so you can help deal with the realities they are facing at home and at work.
Ensure that there are enough materials to complete the restoration, and that there is a plan to replace or replenish items and consumables (cable, poles, terminals, cabinet, splices, tapes, compound) used during the restoration.
Implement a procedure to document all repairs, replacements, and any circuits that were reassigned, which allows any temporary repairs to be made permanent and facilitate reporting for insurance purposes.
Documentation and prioritization of all circuits, and priority circuits, should exist. Priority circuits need to be addressed first in any restoration.
Properly designing and constructing the OSP infrastructure to codes and standards, using premium materials, should prepare it to withstand the event.
Following routine maintenance keeps the OSP infrastructure in the best possible condition as it ages, and allows it to be repaired or restored quickly if disaster strikes.
Lastly, in addition to the emergency restoration plan, keeping your technicians fully trained and having restoration materials readily available aids in shortening the impact and duration of the event.
Do not wait for the disaster to happen! Murphy’s Law implies that if something can go wrong it will, at the worst possible critical location, and at the worst possible time. To prevent Mr. Murphy from having a gleeful grin at our expense, stop him in his tracks by remembering the 5 P’s: Prior Planning Prevents Poor Performance!
Set your plan in place and perform a test run to verify. This allows you to find and correct any errors or omissions in the plan. Is it expensive to do? Absolutely! But is it better to find out before your infrastructure is affected and customers are out of service, or before other tragedies strike? If it is your responsibility, I think you now know the answer.