Drone Swarm: Victor's Work
Tips From Hindsight...
Only after conclusion of this project did the team realize that some of the efforts that were made may not be the best approaches. If you are trying to do a similar project or task, we recommend the following:
- Set up Linux/Ubuntu on a local machine, and use it as a development environment. For PC users, check out Windows Subsystem for Linux (WSL) .You will also need Docker, Python3, cflib, etc. for this project.
- Once you have a working Linux environment, we recommend working everything with Robot Operation System (ROS). ROS will give a more robust and user friendly environment for multi-agent (swarm!) interactions. Check out Crzyswarm.
- Instead of using external cameras for situation awareness, we recommend using the Crazyflie AI deck. It is a open-source package that include hardware and software with a high-performance micro camera, built-in Wi-Fi connection, and onboard computing resources.
- We also recommend starting with the Crazyflie Flow Deck as the primary positioning system. It uses optical sensors to measure its position relative to the ground with high accuracy, and is more robust against external noises and disturbances. Later on if you need absolute coordinates of the drones, the built-in extended Kalman filter can combine the flow deck and the lighthouse positioning system.
Setting up Crazyflie Client
No matter what system and control mechanism you are using, the Crazyflie client can be a useful beginning point and an intuitive GUI to help you understand the basics of Crazyflie drone system, as well as simplify some vital procedures during the project. It is developed with cflib which means theoretically you can use all of its functionalities with python script without loading the GUI.
To setup, first make sure you have a compatible environment. Though in principle the client can be installed and run on many systems, we found that Linux/Ubuntu and windows work the most stable, whereas MacOS makes the client crash. Make sure your systems default python version is 3.x or higher, and any Windows related software supports and is being run on a 64bit version system.
Check out this tutorial on how to install and use the client. Keep in mind that you also need to install the Crazyradio driver if you are using Windows or Linux, which have different procedures depending on your system.
After installation, run the following shell command to start up the client. You may need to add cfclient, which is usually under a library directory of python installation, to the system PATH manually to be able to call this command anywhere in your system
> cfclient
Then, you should see an interface like this:
This means the cfclient is installed the successfully, and from now on you could use it to setup your drones, lighthouse system, or control the drones directly. Check out Setting up Crazyflie and Lighthouse Deck part of our project!
Also, if you wish, there are several tabs about PID tunings, logs, on-board LEDs that are by default hidden. You could bring them out from “View” and gain more control of the drones. More details are in the official tutorial.
Flight Control
Let’s jump through the super basic theories of how things can fly in the first place, like Bernoulli’s principle or Newton’s third law of Motion, and start talking about how drones fly themselves as controlled.
By “drones”, we are actually here talking about unmanned quad-rotor aerial vehicles, which fly just like helicopters but with the following differences:
- Helicopters changes collective/periodic pitch of the rotor blades to control its lift and pitch/roll forces, while drones simply change the power output (or the more technical term Pulse Width Modulation) of its servos.
- Helicopters often require a tail rotor to control its yaw movement, while quad-rotors changes power output of the servos to make unbalanced rotation moments.
One can see from the above figure that a drone have two diagonal propellers spinning in the same direction, and other diagonal two spinning in the opposite direction. This is due to the fact that when a servo rotates, not just the attached propeller would subject to a rotational moment and start spinning, the drone fuselage as well would subject to the rotational moment in opposite direction. If unbalanced, the drone itself would enter a uncontrollably spinning motion. That is why even number of rotors are always needed to be onboard and work against each other.
So,
- To lift, all propellers change power output together
- To pitch or roll, two propellers on one side change power output
- To yaw, two diagonal propellers change the power output
Now that you understand how the drone fly, we can try it out and fly manually on a smart phone (AppStore, GooglePlay)
The above gif is showing the team trying out flying the drone manually, obviously the performance is not sufficient for precision taskings. To operate the drones with enough precision and autonomy, one need to use Crazyflie’s python API cflib .
Here is a official step-by-step tutorial on how to use cflib. Note that the tutorial is using the flow deck expansion package, where as this project is using lighthouse deck as positioning system, though this should not change any significant portion of the process. A typical structure of the code would be something as the follows:
#!/bin/bash
import time
import cflib.crtp
from cflib.crazyflie import Crazyflie
from cflib.crazyflie.syncCrazyflie import SyncCrazyflie
drone_uri= {'radio://0/80/2M/E7E7E7E7E7'}
def Take_off(scf):
cmd=scf.cf.high_level_commander
cmd.takeoff(z=1.0, time=1.0)
time.sleep(1)
def SOME_COMMAND(scf):
cmd=scf.cf.high_level_commander
cmd.go_to(x=1 , y=0, z=0, yaw=0, time=1, relative=True)
time.sleep(1)
def land(scf):
cmd=scf.cf.high_level_commander
cmd.land(z=0, time=1.0)
time.sleep(1)
cmd.stop()
def main():
cflib.crtp.init_drivers()
with SyncCrazyflie(drone_uri, cf=Crazyflie(rw_cache='./cache')) as scf:
scf.cf.param.set_value('commander.enHighLevel',1)
resest_estimator(scf.cf)
takeoff(scf)
SOME_COMMAND(scf)
land(scf)
if __name__ == '__main__':
main()
The above code is a simple example of control the drone to take-off, go to somewhere, and land. Any other tasks can be done with the similar manner.
This code is using the high_level_commander, whereas the official tutorial is using MotionCommander and position_hl_commander. These commanders are APIs tailored for different level of task descriptions — the one used above is more suitable for swarm projects. You can choose which one you prefer from the three and switch in-fly if necessary. Check out here and here for their methods and usage.
The url is the radio channel that a specific drone is using to communicate with the computer, and is used to call to a specific drone. The default factory setting of every drone is radio://0/80/2M/E7E7E7E7E7, but we recommend change this to a more unique one using the cfclient before usage i.e. radio://0/80/2M/E7E7E7E701.
Several points to look out for:
- Keep the code run within the “With” statement. The “With” statement ensures that if the program runs out of it or stops running, the drones will initiate a shutdown process.
- Check the hardware before performing any tasks, before taking off. Make sure that the battery has enough power, the propellers are attached securely and will not fall off during flight, and servos are not fried. Many times the code gives error that seems like a bug, but is actually hardware failures.
- The drone is a hardware that works in physical reality, which means it need time and space to do anything. Use time.sleep() every time a command is sent out to give the drone enough time, and keep enough space around the drone to tolerate any positioning noise of the sensors.
- If the drone crashes, the hardware is designed so that it can absorb the shock and is usually not damaged. But the drones’ firmware will go into a post-crash mode that freeze everything from operating. Restart the drone to reset when this occurs. Also in case that this automation was not invoked in a crash, always prepare an “abort” python script at hand to force stop the propellers.
Finally, if you wish to visualize or record the drones motion data, the appropriate data to keep log for is kalman.varPX, kalman.varPY, and kalman.varPZ. Crazyflie uses extended Kalman filter to estimate its state from any resources it can get (IMU, optic flow, laser positioning, lighthouse positioning, etc.). This means any combinations of sensors can be used together, and all independent observations are incorporated together in the Kalman filter to yield a single output. You can check out details of these variables in cfclient.
Setting Up Experiment Environment
The experiment environment should be set up specifically for the Crazyflie drone system, or more precisely, a safe air space for multiple mini quad-rotors and a suitable geometry for the lighthouse positioning equipment.
The team borrowed a caged space in the lab to fly the drones. This space can ensure the personnel’s’ safety while having abundant attach points for other hardware. When deciding your own experiment environment, be sure to select a place that is
- Large enough
- Safe
- Minimizing disturbances or changes in wind, light, and radio signals
- Keep one side open if you are using a cage that’s similar to ours, as it may create a Faraday cage and disturb the communication signals.
Then we setup the lighthouses of the lighthouse positional system. Follow Setting up Crazyflie and Lighthouse Deck part of the project.
The lighthouse base stations showing above shall be mounted in the environment, and the lighthouse deck shall be mounted onboard of the drones. The base stations emit infrared signals and sweep through the space, whereas the four receivers on the corners of the deck receives geometry information and pushes it to the Kalman filter.
When mounting the hardware, make sure that the base stations are high enough and tilt down a little bit. There should be at least 0.5m(1.6ft) of vertical clearance between the stations and the drones air space ceiling.
Also make sure that the deck is mounted on top of the drones mother board, without anything blocking it.
After the hardware is all setup, the software still need information of the physical world to solve the geometry. This is one of the most significant reason why we need the cfclient, as it has a built-in wizard that can measure everything for us. Click the “Estimate Geometry” button of the lighthouse position tab (might be hidden in “View”) to start this. Refer to this page for more detailed information.
Note that this process need at least 2 base stations to be visible by the drones at any time. But after it is set up, any number from 1 to 4 of lighthouses can be used to estimate positions.
When the geometry estimation concluded, click “write to Crazyflie” to store the information to the drone. This is one of a few information that will be stored in the drone permanently or until overwritten, it will not be reset when reboot the drone. Click “Save configuration” in cfclient to store this information on the computer, so you may “Load configuration” later to other drones that are being prepared for swarm operations.
You should now be able to see something like this:
Note that though this information is permanently store onboard, it will quickly become unreliable or even untrue when:
- any person touched and changed its position/attitude (even just a little bit!)
- the object that it is mounted on was moved (look out for loosen duct tapes etc.)
- the experiment was performed in extended period of time that the light condition is shifted (sunset, sun rise, etc.)
So remember to re-estimate the geometry when the drones flight become unstable, or simply once a couple of hours.
The lighthouse system also have two modes of estimating drone’s position: Crossing Beams, and Raw Sweep. The default setting is Raw Sweep and is working fairly reliable. you could change it by setting the lighthouse.method parameter to your desired value. Check out this page for details about the methods.
Now that we have a working positioning system, let’s click the “Take off” button in cfclient flight control panel to have a hover test!
Swarm Control
Controlling multiple drones as as swarm with python script is very similar to controlling only one drone. Here is an example code structure:
#!/bin/bash
import time
import cflib.crtp
from cflib.crazyflie.swarm import CachedCfFactory
from cflib.crazyflie.swarm import Swarm
factory_settings = CachedCfFactory(rw_cache='./cachea')
drone1='radio://0/80/2M/E7E7E7E701'
drone2='radio://0/80/2M/E7E7E7E702'
drones = {drone1, drone2,}
def Take_off(scf):
cmd=scf.cf.high_level_commander
cmd.takeoff(z=1.0, time=1.0)
time.sleep(1)
def SOME_COMMAND(scf):
cmd=scf.cf.high_level_commander
cmd.go_to(x=1 , y=0, z=0, yaw=0, time=1, relative=True)
time.sleep(1)
def land(scf):
cmd=scf.cf.high_level_commander
cmd.land(z=0, time=1.0)
time.sleep(1)
cmd.stop()
def main():
cflib.crtp.init_drivers()
with Swarm(drones, factory=factory_settings) as swarm:
swarm.reset_estimators()
swarm.parallel(take_off)
swarm.parallel_safe(SOME_COMMAND)
swarm.sequential(land)
if __name__ == '__main__':
main() Note that now each drone has their unique url, and a list of urls are used to define a swarm.
The Swarm object controls drones in a swarm in one of the three methods:
- sequential: the program communicate which each drone one-by-one, wait till one to finish its task before tending to next one
- parallel: communicate with all the drones at the same time, publish commands to everyone in the swarm
- parallel_safe: same as parallel, but will raise exception and reject the commands if any one of the threads/drones give an error
The same command is given to every drone when these methods are called. To control a portion of the swarm, use set_group_mask.
Also be aware that there is no collision avoidance built-in the system, and at this stage the drones cannot communicate with each other to collaborate without computer coordination and human-in-the-loop interference. This is the primary reason why onboard computing and ROS is strongly recommended.
There is an official step-by-step tutorial as well as examples in the cflib Git repository about swarm experiments.
Drone Onboard Computing
If we want the drones to perform more complicated tasks, has better situation awareness, or communicate with other drones, they need a form of onboard computing capabilities. To achieve this, we recommend the use of ROS and Crazyflie AI decks.
ROS is known to have inter-node interaction structure that allows large scale of information being passed between many independent nodes that are working in parallel. This system is very suitable for multi-agent system like ours, with each node being a drone or a device. This would bring collision avoidance capabilities to the swarm.
The Crazyflie AI deck is an open-source hardware/software package with built-in micro camera, Wi-Fi connection, and an onboard CPU. As it is developed by the Crazyflie team specifically for Crazyflie drones, it will work directly on these drones without compatibility issues. Its computation capabilities can boost the capabilities of the system with deep neural networks, reinforcement learning policies, and/or indirect high level controllers. However, the AI deck only work with Linux system. Windows users may check out Windows Subsystem for Linux (WSL).
Model Predictive Control
Until this stage, the swarm system can only carry out very simple tasks. This is enough for the scope of this project, which is to setup stage for future researches with arbitrary tasks. But if we want the swarm to actually carry out those tasks in the future, a more sophisticated system control methods is required. We recommend model predictive control (MPC) and multi-agent reinforcement learning.
By the time this project is conducted, none of the members in the team has education background on MPC, so we cannot provide any detailed information. But if you would like to proceed, we recommend checking out this page together with this paper. This paper also comes with published code that works for both simulation and physical implementation. Below shows a simulation in MATLAB of a slightly modified version of this MPC code.
Author
Victor Xia
This page is authored by Victor Xia and describes work done by him. By the end of this project and creation of this page, Victor Xia is a first year graduate student at MEMS Department, Duke University.
Check out this page for more about him!
Many of illustrations used in this page are sourced from Crazyflie Official Website.
