Investigation and Security Blog

Lessons Learned from the Robot Army

Illustration of multiple robots Implementing lessons learned are key to the successful development of complex systems


My last position at the FBI was Chief of the Technology Innovation Section within the Information Management Division.One of my key responsibilities was overseeing the development of the FBI's new Central Records Complex, a $220 million facility.This project combined a construction project with new as well as legacy IT systems, a high-speed scanning operation, and an army of 114 robots to retrieve physical records within a grid system spanning an area equal to two football fields.

Due to security issues, two systems had to accurately exchange limited but critical messages

A critical linchpin of the system was where two systems met. Due to security considerations, they could not be merged together, but had to pass information back and forth with complete accuracy. Anything less could have had catastrophic consequences for our operations. "Good luck with that!" was the encouragement from one technology executive. To complicate things, these systems were managed by three different components within the Bureau.

Pursue the right solution

All security solutions are a balance of security controls and technical efficacy. Due to some security concerns, our interface solution necessarily leaned heavily towards security controls, requiring us to develop a more precarious technical solution. During one executive meeting as I was briefing this situation, a general counsel attorney stated that the vendor may be at fault for requiring the FBI to develop a complex technical solution in order to integrate the system they were hired to provide.

Several days later, the attorneys who attended the meeting asked to review the contract and set up a meeting with the vendor.

"Nope – please stop." The legal effort was well intentioned, but not going to move us towards our goal of completing the Records Center on time. Introducing legal machinations would only delay or stall the project completely.

We had three types of competing approaches: cyber security, information technology, and legal solutions. After obtaining a variety of perspectives, we balanced these approaches to come up with the best solution for the situation. Don't look for solutions from one type of expert or department. Each expert is likely to be biased towards their area of expertise: The cybersecurity expert will focus on security controls, the information technology specialist will focus on data integrity and availability, and the attorney will focus on holding parties accountable. Focus on what must be accomplished to guide balancing different approaches as you craft your solution.

Ask "dumb" questions

When my team first began testing this interface, the results were not good. Some messages were getting through, others would be held indefinitely in queue or lost completely. Extensive trouble shooting revealed that the problem resided with a "Datapower" configuration on one on the systems. We were facing a tight deadline, and unfortunately no one on staff was familiar with how to reconfigure the device. "We have a couple of contractors looking through the manuals to figure it out." Seeking a better solution, we wound up flying in a technical expert from Florida. As we worked through this issue, I asked my technical lead, "So what does the 'Datapower' actually DO?" A pause ensured – "I'm not really sure."

For several days, I had heard the term "Datapower" over and over, and assumed everyone else knew what its function was. From the context, I had envisioned it being a router of some time. It turned out to be a router/firewall combination.

Nobody is an expert in all aspects of complex systems. Ask questions – you may be surprised that many people will be glad you did.

Thousands of components are part of complex systems. Don't be afraid to be curious about what they do and the role they play in the system.

Breaking through bureaucratic rabbit holes

As we worked to complete the communications interface, our efforts were complicated by the fact that the systems we were tying together were owned by three different entities located in two states. We were facing a tight deadline, and had to navigate three bureaucracies to get our needed configuration changes made. We would contact someone in Entity 1, who would advise we needed to submit a change request. After doing so, we then learned that Entity 2 would need to make a change (necessitating another formal request) before Entity 1 could implement their changes. This circle continued on and on, and time was growing short. The series of emails and online meetings were not moving us forward.

We shifted to having four-hour online workshops with the technical experts from each division. Executive approval was obtained to implement the needed changes, to be followed later with formal documentation. We made the configuration changes during the meetings, so all the involved entities could verify the results and provide feedback real-time. After several weeks of ping-ponging back and forth between the three bureaucracies, we were able to complete the needed changes while maintaining appropriate security measures in a couple of days.

When policies and procedures are preventing you from accomplishing your organization's mission, there is something wrong with the procedure. This often occurs with unique or complex solutions. A waiver of normal practices may be necessary and acceptable so long the intent of the policies is maintained. 

Solutions should prioritize ease of use during daily operations, followed by security and ease of maintenance.

The vendor's solution may not be the right one

The robot system that was integrated into the FBI Central Records Complex is the largest Automated Retrieval and Storage System of its kind in the world. A massive grid system allows the robots to travel above bins stacked 16 high. The grid spans three football field sized fireproof cells – if a fire occurred in one cell, doors to allow the robots to travel between the cells would automatically close to limit the damage. During normal operations, the robots would freely travel among the three massive cells to retrieve the bins containing files within the grid system. The robot would then move the bin to a one of fifteen workstations where a human worker would service the requested file.

During testing, we noticed none of the robots were traveling between the cells. A query to the vendor revealed that the grid had been logically divided into three separate systems to simplify the development of the database supporting the grid operation. This was not how it had been specified – any workstation was supposed to be able to retrieve any file regardless of where it was located in the grid. The vendor resisted changing the system to meet the original specification, saying it was too difficult. However, the "divided grid" arrangement was a poor solution, as it would have required us to assign additional personnel to track file requests. This time I did get the lawyers involved, and the vendor eventually acquiesced and developed an acceptable solution. If technology solutions required additional personnel, they aren't acceptable. Push for the right solution, your team may have to live with it for years or even decades.

Trust your people to work the problem and develop solutions

Throughout this two year process, I was struck with the complexity of this project, and learned to rely on the perspective of the front line workers who worked closest to the various challenges we faced. They were the experts in the daily operations of the center, and I welcomed their expertise and proposed solutions. I knew we were going to be successful when we got to the point where our people recognized potential problems before the occurred and implemented solutions before management was even aware of the potential issue. When the entire workforce understands what the mission is and feels they are contributing part of it, challenging projects are far more likely to succeed.

I am still working on trusting the robots, though. 

List of Audio Surveillance Laws by State
How to Password Protect a MAC Folder

By accepting you will be accessing a service provided by a third-party external to