Giuseppe Pellizzi Prize 2018 28th Members Meeting of the Club of Bologna November 10, 2018 (SPACE RESERVED FOR PHOTO CONCERNING THE THESIS) INTELLIGENT VISION SENSING SYSTEMS FOR PRECISION AGRICULTURE PRACTICES IN FLORIDA CITRUS PRODUCTION Daeun Dana Choi, Ph.D. dxc519@psu.edu Assistant Professor, Dept. Ag and Bio Engineering Penn State University, Pennsylvania, U.S.A.
BIOMECHATRONIC ENGINEERING B. E. at Sungkyunkwan University, Seoul, South Korea Summa Cum Laude Research Intern: Intelligent Robotics & Mechatronics System Lab ECONOMICS B. A. at Sungkyunkwan University, Seoul, South Korea Study Abroad, Vrije University, Amsterdam, Netherlands AGRICULTURAL AND BIOLOGICAL ENGINEERING DAEUN DANA CHOI, PH. D. Education and Experiences M. S. and Ph.D. at University of Florida, Gainesville, FL Research Assistant in Precision Agriculture Laboratory Master Thesis: Detecting and counting citrus fruit on the ground using machine vision Dissertation: Intelligent sensing systems for precision agriculture practices in citrus industry AGRICULTURAL AND BIOLOGICAL ENGINEERING Assistant Professor at Pennsylvania State University, University Park, PA, U.S.A. Lab Supervisor at Smart Agriculture Laboratory Photo credit to Emiel Molenaar
Intelligent Vision Sensing System for Florida Citrus Production slide #3 Introduction Production (1000 tons) Orange production in top five leading countries during 2011-2016 21000 18000 15000 12000 9000 6000 3000 20482 8166 6900 6023 3666 8560 16361 7501 7000 5890 4400 8699 17870 6140 7600 6550 4533 9389 Citrus Production Decline in the United States 16716 5763 6600 5954 4515 9392 14320 5362 6900 6241 4400 2011-12 2012-13 2013-14 2014-15 2015-16 Year Brazil United States China Source: Citrus: world markets and trade, USDA Foreign Agricultural Service (2017) 9852 Citrus? High-valued specialty crops in international trade markets Fresh market Processed Product Citrus Production 117.3 million tons per year (2007-2014) Major crop: Oranges (69.2 million tons per year, 59%) Source: Citrus fruit statistics 2015, Food and Agriculture Organization of the United Nations (2016) Dana Choi Intelligent Vision Sensing System for Florida Citrus Production 3
Intelligent Vision Sensing System for Florida Citrus Production slide #4 Challenges in the Florida Citrus Industry Orange production in the United States Florida Citrus Industry Incomparably dominant producer more than 80% of the oranges in US Economic impacts $8.9 billion in the 2007-2008 season $10.8 billion in the 2012-2013 season Orange production decrease Production dropped to 73% in the 2004-2005, 68% in 2014-2015 and 59% in 2015-2016 Orange Production (1000 tons) 20000 15000 10000 5000 0 Florida California USA 1995-1996 Canker & Hurricanes 1997-1998 1999-2000 2001-2002 2003-2004 Year 2005-2006 2007-2008 Widespread of HLB 2009-2010 2011-2012 2013-2014 Source: Florida citrus statistics 2014-2015, USDA-NASS (2016) Dana Choi Intelligent Vision Sensing System for Florida Citrus Production
Intelligent Vision Sensing System for Florida Citrus Production slide #5 Precision Agriculture Optimization of Farming Input Efficient farming by site-specific crop management Increase crop yields Maximize profit for growers Reduce environmental impact Citrus grove in Lake Wales, Florida Intelligent Sensing System Automatic data acquisition + machine Intelligence Crucial for agricultural automation, robotics, precision agriculture management Reduce production costs and increase productivity Dana Choi Intelligent Vision Sensing System for Florida Citrus Production
Intelligent Vision Sensing System for Florida Citrus Production slide #6 Intelligent Vision Sensing to support precision agriculture in Florida citrus production 01 02 Early Stage of Crop Development Immature citrus detection system for yield forecasting using a Kinect depth sensor and a gradient vector based method Pre-harvest to Harvest Premature citrus fruit drop detection using regular RGB cameras and an adaptive image correction algorithm 03 Post-harvest Precise citrus defect inspection using regular RGB cameras, deep learning, and parallel computing techniques Dana Choi Intelligent Vision Sensing System for Florida Citrus Production
Intelligent Vision Sensing System for Florida Citrus Production slide #7 Immature Citrus Detection System for Yield Forecasting Dana Choi Intelligent Vision Sensing System for Florida Citrus Production
Intelligent Vision Sensing System for Florida Citrus Production slide #8 Immature Citrus Detection for Yield Forecasting Forecasting citrus production Early decisions for planning operations and marketing Estimating the number of fruit by manually surveying sampled areas Difficult to produce consistent, accurate and timely agricultural market data and forecasts Accurate estimation of fruit count in individual tree Kinect V2 Truck Hand-held Golf cart Dana Choi Intelligent Vision Sensing System for Florida Citrus Production
Intelligent Vision Sensing System for Florida Citrus Production slide #9 How many citrus can you find? Dana Choi Intelligent Vision Sensing System for Florida Citrus Production
Intelligent Vision Sensing System for Florida Citrus Production slide #10 Answer: 18 citrus False positive warning area! Dana Choi Intelligent Vision Sensing System for Florida Citrus Production
Intelligent Vision Sensing System for Florida Citrus Production slide #11 Behind the scene Near-Infrared Image Computers may see better than human does. Dana Choi Intelligent Vision Sensing System for Florida Citrus Production
Intelligent Vision Sensing System for Florida Citrus Production slide #12 Gradient Vector Field on the Citrus Surface ügradient Vector of depth (D) V = D = D x, D y = D *, D, üdirection of the Gradient Vector v = V V = D D = 1 D D *, D, Depth map of a citrus surface Dana Choi Intelligent Vision Sensing System for Florida Citrus Production 12
Convex function Concave function 2D Representation of 3D Surfaces ü Divergence: the amount of spreading vectors Div v = 3 v = x, y 3 1 D D *, D, = 4 56 ( 86 9 8* + 86 ; 8, ) Divergence
Intelligent Vision Sensing System for Florida Citrus Production slide #14 2D Representation of 3D Surfaces Convex function Rotating vectors in 2D GVF plot in clockwise direction with fixed origin. ü Vorticity: the amount & direction of microscopic circulation ω = v = x, y, z D *, D,, D B Concave function = 86 ; 8* 86 9 8, k ω = D, x D * y v = D *, D, v = D,, D * Vorticity Dana Choi Intelligent Vision Sensing System for Florida Citrus Production 14
Intelligent Vision Sensing System for Florida Citrus Production slide #15 2D Representation of 3D Surfaces Convex function Concave function v = D *, D, v = D,, D * Vorticity Dana Choi Intelligent Vision Sensing System for Florida Citrus Production 15
Intelligent Vision Sensing System for Florida Citrus Production slide #16 Characteristics of the Citrus Surface Contour map of the citrus surface v = D *, D, v = D,, D * Dana Choi Intelligent Vision Sensing System for Florida Citrus Production 16
Intelligent Vision Sensing System for Florida Citrus Production slide #17 Characteristics of the Citrus Surface 2 1.5 1 0.5 0 0.5-1 -1.5-2 Contour map of the citrus surface Divergence Vorticity Dana Choi Intelligent Vision Sensing System for Florida Citrus Production 17
Intelligent Vision Sensing System for Florida Citrus Production slide #18 Spherical Object Detection Algorithm 2 1.5 1 0.5 0 0.5-1 -1.5-2 Condition 1 The center pixel of a sphere has the maximum divergence value among other pixels Condition 2 All pixels of a sphere maintain non-negative divergence and vorticity values Divergence Vorticity Dana Choi Intelligent Vision Sensing System for Florida Citrus Production 18
Intelligent Vision Sensing System for Florida Citrus Production slide #19 Classification (SVM and Random Forest) Feature description Number of features Roundness 1 Number of edges/area in NIR 1 Mean NIR 1 Mean divergence 1 Mean vorticity 1 Mean divergence after Gabor filtering 8 Mean vorticity after Gabor filtering 8 Mean magnitude of gradient vectors after Gabor filtering 8 Mean of synthetic texture after Gabor filtering 8 Total 37 Dana Choi Intelligent Vision Sensing System for Florida Citrus Production
Intelligent Vision Sensing System for Florida Citrus Production slide #20 Results Result will be displayed on RGB Image or Near-infrared Image for comparison Day Image Image with Strong Sunlight Statistics Night Image with an External Light Correct Identification = Correctly Identified Citrus Fruit/Actual Number of Fruit False Positives = Incorrectly Detected Object/Actual Number of Fruit Night Image with a Partial External Light Dana Choi Intelligent Vision Sensing System for Florida Citrus Production
Night image
Night image
Intelligent Vision Sensing System for Florida Citrus Production slide #23 Daytime image Dana Choi Intelligent Vision Sensing System for Florida Citrus Production
Intelligent Vision Sensing System for Florida Citrus Production slide #24 With strong sunlight Dana Choi Intelligent Vision Sensing System for Florida Citrus Production
Intelligent Vision Sensing System for Florida Citrus Production slide #25 With strong sunlight Dana Choi Intelligent Vision Sensing System for Florida Citrus Production
Intelligent Vision Sensing System for Florida Citrus Production slide #26 Results Using all 37 Features Actual # of Fruit Random Forest Support Vector Machine CI (%) FN (%) FP (%) CI (%) FN (%) FP (%) Day 335 297 (88.7) 38 (11.3) 23 (6.9) 275 (82.1) 60 (17.9) 15 (4.5) Night 305 274 (89.8) 31 (10.2) 17 (5.6) 278 (91.2) 27 (8.9) 15 (4.9) Sum 640 571 (89.2) 69 (10.8) 40 (6.3) 553 (86.4) 87 (13.6) 30 (4.7) Selected Features using Random Forest Actual # of Fruit RF Top 5 features RF Top 3 features CI (%) FN (%) FP (%) CI (%) FN (%) FP (%) Day 335 292 (87.2) 43 (12.8) 26 (7.8) 281 (83.9) 54 (16.1) 38 (11.3) Night 305 265 (86.9) 40 (13.1) 21 (6.9) 264 (86.6) 41 (13.4) 21 (6.9) Sum 640 557 (87.0) 83 (13.0) 47 (7.3) 545 (85.2) 95 (14.8) 59 (9.2) Dana Choi Intelligent Vision Sensing System for Florida Citrus Production 26
Intelligent Vision Sensing System for Florida Citrus Production slide #27 Unit:mm Dana Choi Intelligent Vision Sensing System for Florida Citrus Production
Acknowledgements Dissertation Supervisors Dr. Won Suk Lee Dr. John Schueller Dr. Reza Ehsani Dr. Fritz Roka Dr. Corey Toler-Franklin Dr. Dorota Haman Dr. Mark Ritenour Dr. Alireza Pourreza Mr. Hao Gan Dr. Yao Zhang Mr. Chenglin Wang Dr. Kezhu Tan Mr. Arumugam Kalaikannan Mr. Mike Zingaro
THANK YOU Thank you for coming Have a nice day Image source: Orange tree, oil painting by Alan Flood