Today I thought I would pull together some of the information that has been produced as a part of my GRDC sponsored Nuffield Scholarship on the use of unmanned aerial vehicles in the grains industry. It has been an amazing experience and I encourage farmers from around the world check the Nuffield organisation to see if a scholarship would suit them.
Traveling through the USA & Canada as part of my Nuffield Scholarship (Thanks to GRDC) I have heard the word data more times than I could count. In this post I am going to remove the platform aspect of the unmanned system and focus on sensors and the data they provide. Then in a later post how we can use it. In this article I do plan to try cover some concepts not widely discussed in the current UAS environment.
The reason an unmanned system is flown is to collect data, then turn that data into information to help monitor, assess and ultimately make timely, cost effective decisions based on that information. When collecting data it needs to be good quality. It is important not to confuse data quality with data type. For example, many tend to gravitate straight to the amount of megapixels a sensor captures, neglecting its spectral accuracy.
If we consider what our target is when collecting data from a UAS in a grain farming situation, it will most commonly be vegetation (not always but let’s focus on that). Collecting spatially referenced data of vegetation is by no means a new endeavour. This information has been collected as vastly as Landsat satellite imagery and as specific as a GreenSeeker. Generally, for vegetation, similar bandwidth reflectance is measured irrespective of proximity to the target. The same is true for sensors used in UAS. Why is this the case? Well you can read the long answer here (Remote Sensing of Biomass). The short answer is photosynthesis. In a plant that is photosynthesizing, the chlorophyll will absorb large amounts of ‘visual’ light, particularly blue and red, and reflect near infrared (NIR) light. The more photosynthetic activity, the more NIR light is reflected and less visual light absorbed. Conversely, inactive vegetation will reflect more visual and less NIR.
The best contrast is between red and NIR light which is what is generally used when calculating the Normalised Difference Vegetation Index (NDVI). NDVI is a good indicator of plant health and measure of biomass. Consequently, most sensors used to determine NDVI look into the red and NIR bands – some more accurately than others. The chart below shows the reflectance curve of green grass over different wavelengths. Below the X axis is a rough spectral guide to some of the well-known sensors available to us.
What is most notable is the wavelength spectrum or band at which each of the sensors read. If we consider the GreenSeeker, it is extremely specific at capturing a certain wavelength in the middle of the red spectrum and similarly specific in the NIR. At the other end of this comparison you can see that the S100 modified camera has very broad spectrum for each channel that it reads. Consider (what was before modification) the S100’s ‘red’ channel which reads roughly from 0.67um to 0.76um with the modified filter. Post modification, this channel is renamed NIR and measures reflectance in an area that covers red right through to NIR. The S100 modification retains the blue and green channels which replaces red when calculating NDVI. Another significant point that this chart does not show is the interference that can occur between the different bands in a point and shoot camera. Check out the S100 reflectance chart about half way down the page in this link which shows some NIR response in the blue and green channels.
It has to be noted that it is hardly fair to compare the S100 and GreenSeeker in a practical sense for several reasons with the main one being that you would not mount a GreenSeeker on a UAV as it needs to be close to the target (the GreenSeeker is an active sensor meaning that it emits light and measures that reflectance, the S100 is a passive sensor just reading reflectance from the sun). In addition, the GreenSeeker measures only one point whereas the S100 collects data on 12 million pixels. The reason I do compare them is because they can both be used to produce an NDVI map. In fact despite the spectral differences from each of these sensors and the proximity to the target, RoboFlight claim from their tests that NDVI data collected from a GreenSeeker and modified S100 correlate in a linear fashion very closely (r squared > 0.9). So we know that the two sensors correlate well but the correlation will never be a fixed formula because sunlight reflected will always be different based on sun angle, atmosphere conditions, cloud etc. The S100 and GreenSeeker would probably work best as tools which complement each other. For example, map a large area with the S100 on a UAS. The resulting dataset could be calibrated using GreenSeeker data collected in the field at the same time as the flight. Potentially if S100 data is always calibrated against the GreenSeeker, inter-paddock and inter-season comparisons can be made.
We are starting to see a new wave of sensor development designed specifically for UAS and agricultural and environmental industries. An available example is the Airinov Multispec 4C. This sensor captures 4 distinct, narrow spectral bands with no interference. These bands include green, red, red edge and NIR. What makes this package special is that not only does this sensor package look down at the vegetation; it is also looking up measuring sunlight with a lux meter. This should allow us to generate data that can be calibrated without the need for ground truthing with a GreenSeeker or similar ‘active’ device. Another feature of this sensor is that it uses a Global Shutter which means all pixels in a photo are captured at exactly the same time eradicating any motion blur. The 4C has much less spatial resolution than the S100 (1.2MP vs 12MP). Expect to pay over US$10,000 for this sensor package, not including UAV or processing software.
In summary, this article aims to explain how there is more to a UAS sensor than just megapixels. It is important to understand the spectral response of vegetation and how this can impact your sensor choice. A modified Canon camera such as the S100 is a great option for UAS but its limitations must be understood. Work needs to be done to analyse the results and accuracy of the new sensors such as the Multispec 4C.
* Further notes: The most common sensor used in small UAS (mid 2014) is a Canon S100 or similar variant. This camera was never designed to be flown in an UAS but the internal GPS, fast shutter speed (1/2000), relatively large sensor (12.1 MP 1/1.7″ Canon CMOS), low weight (198g), ability to be modified to detect NIR, CHDK compatibility for intervalometer, and low cost (<$500) all contribute to a well suited sensor for this application. Flown at 120m this camera can provide a ground resolution of 3.5cm.
This week I had the privileged to attend two conferences in the US focusing on unmanned aerial systems (UAS) in agriculture. The first was the Delta AgTech Symposium in Memphis, Tennessee and the second was the Precision Ag Aerial Show at Decator in Illinois. These conferences are the beginning of my private study as part of my Nuffield Scholarship, thanks to my sponsor the GRDC.
As a forward to this discussion, I do focus more so on the fixed wing type UAS for large area mapping – see why here – and therefore generally do not comment on multi-rotor systems, GoPros, First Person View (FPV) etc even though these were covered at these conferences.
Both conferences were a combination of educated/independent speakers and vendors. There was a couple 1 hour multi-rotor demonstrations at Delta AgTech and two full days of demonstrations of fixed wing and multi-rotors at PAAS. The audience was extremely diverse – from curious farmers, agronomists, all the way to sensor & software companies and interestingly a large presence from insurance companies.
Themes I picked up on:
Everyone agrees that data needs to be more than pretty pictures – it needs to be actionable. Futhermore, boots still need to be on the ground, ground truthing data, explaining WHY. No one is claiming UAS replaces clever farmers and informed agronomists.
There is broad acknowledgement that UAS in agriculture needs to show a return on investment but only one case study was mentioned with actual figures.
Data processing is an issue. Some are claiming the solution is server based i.e. ‘in the cloud’ but acknowledge internet bandwidth is a bottleneck.
Sensors are a hot topic. Most manufacturers claim they have vastly improved sensors coming. Talk at the moment is around affordable true multispectral combined with irradiance/incident light measurement.
We were reminded a couple times not to dismiss satellite imagery as an option. Satellite sensors are always improving and getting cheaper. Will we have 10cm GSD from multispectral satellite in 10 years?
Generally frustration with the FAA on the time it is taking to develop rules around UAS. In saying this, there is acknowledgement that rules are needed.
Some other interesting points picked up over the last week:
We are starting to see some UAS companies offering early versions of vectorization (i.e. points, lines and polygons) in their server solutions using imagery collected via UAS with the main example variable rate application map of fertilizer in crop based on NDVI. This is not new technology – it has been done with satellite imagery for years.
With the speed of technology rapidly changing are we considering the upgrade path of current UAS? Can the sensor and GPS be upgraded without buying a whole new system?
Most of the commercial grade fixed wing UAS seem to be able to fly well, have autopilots that just work and acceptable ground control software. Product differentiation will probably be around sensor integration and options, support, innovation and whole system workflows including speed of data processing. This is just a snapshot of what was covered in the last week. I will be compiling a report that encompasses my whole USA/Canada trip which will be available towards the end of the year.
Technology in farming is constantly evolving. Collecting accurate, reliable georeferenced (location in terms of GPS coordinates) data is essential to capitalise on technologies such as variable rate application of chemicals and fertiliser and aid in crop monitoring at a level once not imagined. Some current forms of collecting georeferenced paddock data include:
Combine harvester – yield maps (crop yield as harvester works through paddock)
Satellite imagery – colour and near infrared (NIR) bands to produce natural images & vegetation indices such as Normalised Difference Vegetation Index (NDVI)
Aerial imagery – similar to satellite but offers higher resolution at higher price & some other sensor options
Tractor – Greenseeker (plant biomass), digital elevation model (DEM) collected from high accuracy GPS
Utility vehicles e.g. Soil sampling pH & nutrition, electromagnetic conductivity, Greenseeker, DEM
Handheld with GPS – Greenseeker, soil sampling
Stationary – moisture probe, weather station
Unmanned Aerial Vehicles (UAVs) are emerging as a cost effective way to collect data with many advantages over the traditional forms listed above. UAVs are as the name suggests an unmanned vehicle which flies over the paddock to collect data. These machines are generally compact, can be cheap, mechanically simple, fly below cloud cover and are on there way to being easy to operate with advanced autopilot systems.
Over the last 6 months I have begun researching civilian UAVs and their application in agriculture as part of my Nuffield Scholarship. Furthermore, I have been testing a budget UAV platform which I will discuss in a later post. The aim of this post is to aggregate key information and ideas on the topic into one space. It is by no means comprehensive – more of a beginning. Note that I am not a pilot or lawyer. This article is general in nature and does not give permission to fly or legal advice. Lets start with a sky-high view.
The Agricultural UAV Solution
It is important to consider all aspects pertaining to the agricultural UAV (aUAV) Solution which I define as a robust, timely, cost effective way to collect usable data to improve yields and overall profitability in sustainable farming systems. Consider the following formula:
All components of the formula need to be working well and working together for the product to be successful technology. Now enough of inventing acronyms and formulas that will inevitably change, it’s time to flesh out the components of the aUAV Solution.
There are two main platforms available: fixed wing and multi-rotor. A fixed wing platform has the advantage of covering large areas efficiently, whereas a multirotor shines in being able to remain very stable in challenging conditions with large payloads.
Due to the scale of broadacre grain growing in Australia, my interest lies predominately with the fixed wing platform type, as paddocks often exceed 250ha (~620ac). ConservationDrones has an excellent list of budget fixed wing platforms they have used as an example.
Global Positioning Systems (GPS) are the backbone of most spatial technologies. GPS on the UAV tells the autopilot where it is at all times. In addition, GPS links the data collected to it’s spatial position (aka geo-referencing).
Many UAVs are equipped with a u-blox GPS receiver or similar which is compact and provides <5m horizontal accuracy. These systems are affordable and are accurate for most situations.
An exciting development is the Piksi by Swift Navigation, which is a low cost Real Time Kinetic (RTK) GPS receiver that promises to sell for around $1000 which is unheard of in the world of GPS. The Piksi offers centimetre level accuracy inside a compact design ideal for small UAVs. The improved accuracy will be invaluable for autonomous landings and improved accuracy of geo-referencing data.
We are seeing UAV autopilots improve very quickly with increased reliability, especially within the open source community. Autopilots are essential for being able to effortlessly fly over a whole area to collect the desired data. DIY Drones‘ APM:Plane is often the autopilot of choice for hobbyists and entry to mid level platforms. It uses the same hardware and similar software to the APM:Rover I built last year.
There are several other autopilots available, commercial and open source, that are worth checking out. Google it.
Usually the UAV is communicating with a ground control station (GCS) via radio link. GCS is usually just a laptop computer with software such as Mission Planner. Mission Planner is also used to set the flight paths for the UAV missions.
The most complex part of collecting good data is having the correct sensor. For plant biomass data, the most important spectral range is in the near infrared spectrum. The two most common options include Tetracam ADC Lite built specifically for UAVs or a digital camera modified to capture within this spectrum (MaxMax for example). The latter option is the most cost effective solution. Some preliminary studies show that some good results can be achieved.
Researchers are working hard improving sensors for UAVs. For example, TerraLuma, is a research group at the University of Tasmania. Projects of interest include high accuracy geo-referencing of imagery ‘on the fly’ and the use of a hyperspectral pushbroom scanner to collect data.
Public Lab (an open source community) is also working at modifying cameras similar to MaxMax but also on cheaper devices such as web cams. The recently achieved funding through a Kickstarter campaign. Maybe we will have another cost effective solution soon. See also Pi NoIR.
It is worth mentioning that it is very common for UAVs to have a GoPro camera (or similar) mounted to capture high definition video footage. This video footage is valuable for visually monitoring crops from the sky but is generally not processed to geo-referenced data. There is always exceptions such as shown in this video over a construction site where video footage is used to generate a 3D model.
Data Processing & Integration
Although collecting good data is the most challenging part, the most time consuming (and/or expensive) part can be processing it to a point where it can be integrated into precision agriculture systems. Generally the UAV will follow a lawn mower track collecting an image images at a defined interval with a generous defined overlap. The raw data will usually be images (up to several hundreds – think gigabytes) with a single GPS position and maybe a bearing per image. The challenge for the data processing is to stitch these images together to generate one homogenous data set. Every image is affected by the differing roll, yaw and pitch of the UAV as the image is captured. Some of the more common applications include:
Drone Mapper is a successful web based startup which effectively filled the affordable yet professional data processing gap
Microsoft ICE is a web based and free to use but only stitches images without offering geo-referencing or 3D modelling unlike the above mentioned applications
VisualSFM, CMVS, and CMPVMS – Flight Riot does the hard work explaining how to use this software to generate a 3D model from digital camera photos. This is probably one of the more complex processes but uses all free(ish) software.
Once a geo-referenced, homogenous, data set over a paddock is achieved it could possibly undergo further post processing to determine NDVI. This raster data may then, for example, be used to define zones for in crop variable rate fertiliser application.
As mentioned, some of the above software is able to create 3D models from 2D photographs. These 3D models could be used to create digital elevation models (DEM) which is valuable in farming for determining water movement.
Legal & Operation
In Australia, the Cival Aviation Safety Authority (CASA) rule the sky. CASA has rules governing the use of UAVs (which they call unmanned aerial systems or UAS) and is in the process of re-evaluating some regulations. See a summary of a recent speech from CASA here.
To operate a UAV/UAS commercially in Australia you need to have a certified operators certificate. A list of those certified is available here.
CASA have done well to have a system set up for UAS. The USA is lagging behind and is just now establishing rules and regulations are UAVs.
You can buy a calibrated, tested, ready to fly system built from budget readily available components and open source autopilot. For example Event38 and Flight Riot.
The third option is to go fully DIY. I have tried this using a Finwing Penguin fixed wing platform, APM:Plane autopilot, ordinary Canon digital camera as sensor. I am yet to process any images into geo-referenced datasets. I will post more about this soon. Here is an image from one of my first flights.