DroneNet: A networking application for a robotics usecase

Preface

If you try to visualize a digital communication network, what do you envision? It is not a physical entity and although we cannot see packets entering and exiting the ports on our device, we have similar concepts of flow in other domains. You can draw a parallel to blood cells traveling through our veins or to the cars speeding along a highway. You can make a sketch in your mind that will define how data travels and it will help you understand the system. In the future, it may help you explain certain concepts to somebody else in an easily accessible way. A completely new concept wrapped in a familiar idea is easier to unwrap than an unfamiliar gift in an unfamiliar package.

For me, it's not the image of the familiar that rooted itself in my memory, but of something otherwordly. The words of Murakami, when he talks about small magical creatures at work on their craft, draw an imaginative metaphor for developing a network. The creatures "... climbed down from the bed to the floor, and from under the bed, they pulled out an object the size of a Chinese pork bun. Then they sat in a circle around the object and started feverishly working on it. It was white and highly elastic. They would stretch their arms out and with practiced movements, pluck white, translucent threads out of the air, applying them to the fluffy, white object, making it bigger and bigger. The threads appeared to have a suitably sticky quality..." I can see the description of network engineers physically putting up connections as the creatures did. The author also introduces the perspective of a new member of the community. "Once the girl begins to get the hang of it, plucking threads out of the air is not too difficult. She has always been good with her hands, so she is able to master this operation right away. If you look closely, there are lots of threads hanging in the air. You can see them if you try." I wouldn't describe network engineering as easy to master as plucking threads, but they are a communication lifeline, and being unaware of them is ignorant. This is what I learned after taking the Network Systems class, which ultimately began the project I am about to document here.

Introduction

Being a roboticist, I chose to tie my Network Systems project to a physical component. If not for professional purposes, then for the purposes of adding a realistic component to the system. It was a project aiming to create a communication system between a group of drones. We designed the network protocol in a decentralized fashion to play to the advantages of small replaceable system components. The communication architecture was implemented as a minimum viable product, however, this project can be built upon for more complexity and realism. Not the magic realism that I mention in the Preface, but the real sort of additions that could scale to an application. I describe the process of the multi-layers architecture below. You can also find the code on my github page. It would be a blast if somebody chose to spin something out of this project.

What is the application?

We define the system as one where a group of drones is released into the field. The goal of the drones is to capture images of the terrain and send the information back to the ground station (GS). The photos are collected at the base station and analyzed by a neural network (YOLOv5 or Faster-RCNN) to extract features relevant to the mission. For example, in traffic control systems, images of cars will be segmented while for search and rescue systems, instances of people will be counted. For disaster response, direction of fires or floods can be extracted. For agriculture, crops can be monitored. The drones have to capture high-resolution images. Only one drone is connected to the GS. The rest of the drones have to transfer images from one to another to pass them to the GS.

Drones are easy to deploy and have low acquisition and maintenance costs. However, drones are limited in their system resources. They generally have low battery levels and weak onboard processing power. To increase the speed and the quantity of data transferred from each drone, a networking scheme has to be developed to increase communication efficiency between the drones. The communication and networking design has to be robust and adaptable to dynamic network topology.

Justification

We use the Phantom 4 Pro V2.0 drone as a reference for drone specifications. The relevant specifications are shown below.

Max speed 45 mph or 20m/s
1-inch CMOS camera sensor
Effective pixels: 20M
Lens has FOV 84° 8.8 mm/24 mm (35 mm format equivalent)
microSD max Capacity: 128 GB
Supported photo formats JPEG, DNG (RAW), JPEG + DNG
Image size: 3:2 Aspect Ratio: 5472×3648

From the paper by Mittal et al., we learn that to apply deep learning to images, the drone has to be somewhere between 10-70m in altitude. For the purposes of our project, we use 30m because it is enough to capture images of small objects, such as bottles, human actions, and vehicles. To calculate the network requirements we use the relation between focal length and sensor width is proportional to flight height and observed ground area. We find out that:

9MB/2.2s = minimum 4MB/s requirement if the drone flies at max speed 20m/s
At 20m/s, the drone can cover ~500m sq.
One drone can map 300 acres in 60 mins.
10 drones can cover 300 acres in 3.6 mins.
300 acres is about the size of the CU Boulder main campus

The above calculation is from maximum optimism. A true drone fleet will not be able to yield this performance of 300 acres. An interesting report for those invested in the field: how many drones, how many acres, how much time?

Proposed Solution

The complete architecture is a collection of threads that share information via JSON files. The Figure below shows the components which run simultaneously as soon as the user starts our program. The Receiver spawns as many threads as it needs to receive images from any neighbors that want to connect. It receives the images and stores them locally. The Sender reads the routing information, checks to see if there is a possible route to the destination drone, and tries to connect to the next-hop drone. The Discovery and Routing algorithm is split into two threads. The Client initiates the neighbor discovery by broadcasting a hello message. The broadcaster server listens to these broadcasts and responds with its routing table and other information (its ID and IP). The client then uses a flavor of OSPF and link-state to update its routing table.

The Figure below gives an overview of the threads and how they communicate information with each other. They use JSON files to store and retrieve information. A drone IP table is a dictionary from drone ID to its IP. The virtual switch tells drones which neighbors they can speak to and which ones they cannot speak to. This is a software simulation of a real switch. The routing table keeps track of drones in its network, next-hop, and the cost to get to them. Finally, the image list is just a set of images in a folder that can be used by the sender and receiver to store and retrieve images.

The algorithm implemented for routing the image transfer over a network of drones as explained above has been tested with various scenarios. Instead of testing with real drones, we would spawn each process in a virtual machine.

Linear configuration of drones.
Linear configuration of drones, where a drone disappears.
Linear configuration of drones, where a drone appears.

Certainly a limited set of scenarios, but a terrific starting point for a real-life application.

Conclusion

We developed and tested a network topology that allows multi-hop image transfer. The generality of the program lies in its ability to adapt to changing topology structures. It can handle drones that appear, disappear, or change configuration. In a real-world scenario, this means that technicians do not have to be present to tackle any connectivity problems. They can remove a drone from the configuration to spawn a new one without having to adjust the IP address or routing algorithm. This ease of use would be very valuable on the field and something we aimed to do with this project.

Although we completed a basic implementation, there are many more issues that can be considered. One is that the cost is currently only a reflection of the number of hops. If one drone has two neighbors, then these neighbors are identified as candidates for image transfer. This is not practical because the strength of the connection between drones varies significantly depending on distance or even their network modules. The cost should reflect these differences. We also did not tackle the problem of security. These drones should only communicate with one another and not be influenced by malicious hackers. We also did not tackle the count-to-infinity problem which is a known issue with the flavor of link-state routing that we have implemented. Our testbed could have been significantly improved. We only got a chance to test our program with the various laptops that were available. However, it would be beneficial to test the robustness of our code using virtual machines and a real virtual switch. This switch would be able to physically disconnect various drones and allow us to test a more realistic scenario. We also do not take into account network stability, packet loss, or ACKs. For example, one drone may be transferring an image and then going out of the range of its neighbor. In this case, the drone needs to store the image partially and resume again when the connection is re-established. This would be an important consideration for environments in unstructured terrain or those with many obstacles that may force the drone to fly out of range to avoid them.

Overall, the project was interesting and complex; we learned a good deal about these kinds of decentralized ad-hoc networks that could be used on drones or cards, or other multi-agent systems.

Thanks for reading and take a look at the following repo for a more detailed implementation analysis: https://github.com/nataliya-dev/drone-network