SiteSee Accuracy

At SiteSee, a common question asked of us is "what is the accuracy of your photogrammetry model?".  For tower infrastructure owners & co-locating tenants concerned with accuracy of information - this is a justified question.  This article outlines SiteSee's methods in answering this question.

It is important to note that this article is comparing accuracy not precision. For a good explanation of the difference between accuracy and precision please click here to for a detailed article written by James Dunthorne. To summarise, the accuracy is the difference of the means, and the precision is the standard deviation of the measurements.

While the question seems simple enough, it is difficult to answer. I was expecting a simple answer like “+-Xmm”. It is not that simple. Accuracy requires a reference or known measurement for comparison. This is a problem, as we are measuring real world infrastructure in the real world environment. Real world means the wind moves the structure, the sun heats and expands the steel structure - thus changing the dimensions. Hence the expected variation would limit the overall accuracy of any measurement method.  

This experiment is not meant to be an exhaustive exercise in surveying. But rather a short, sharp determination of the suitability of the SiteSee photogrammetry model’s accuracy compared to the industry bench mark of a Lieca Laser Scan.

The results are surprisingly close given the laser scanner is $140K compared to the photogrammetry capture equipment costs of $7K.

This fundamentals of the experiment is to measure between two elements of a Telco lattice tower with a Laser Scanner and compare it to a model prepared by SiteSee using Photogrametry, UAV and GCP. The reference measurements being used are generated by a Lieca Laser Scanner, which is considered to be a good surveyor quality instrument.

The two model elements being measured are;

a) underside of a gantry to, 
b) the underside of the steel resting on footing.

There are three dimension of each point, however this experiment is concerned with the major dimension of the telco tower, the vertical or 'z' dimension.

 Overall Lidar Scan

Overall Lidar Scan

Target: Telco Tower, Pullenvale Brisbane Australia
Lidar Acquisition:  Leica P40, unknown number of scans, .pts file.
Photogrammetry: DJI Phantom 4Pro, 5xPropeller Aeropoints, Context Capture v4.4.8.561.
File Type / Output: .obj mesh model.
Measurement software: CloudCompare V2.9beta.

Background to Data Acquisition:
The tower was flown with a DJI phantom P4 Pro, using SiteSee's purpose built TowerCapture app developed in cooperation with Drone Harmony. Propeller Aero points were used to for Ground Control Points (GCP). The flight plan follows SiteSee's proprietary vertical structure flight plan. Approximately 1500 images were captured. ContextCapture was used to perform the photogrammetry. No manual tie points were used, just the GCP. The resulting model was a mesh in OBJ format with texture.

The lidar was captured by CR. Kennedy using the Leica P40. It is unknown how many scans were used. The resulting point cloud was provided in a PTS format.

The procedure of measurement was very simple. Select a point in lidar point cloud/mesh using the CloudCompare measure tool, followed by the second point. However, it became evident that identifying the physical edge in the model is far from simple. There is no clear definition of an edge when zooming into the target area. The underside of the gantry becomes a blur of points. This became the first point of error. Edge selection. Rather than performing any post processed edge detection or filtering, this experiment is using the points as they are in the model. It is hoped that it demostrates the complex nature of describing accuracy in a real word model capture.

 Bottom edge of gantry point cloud

Bottom edge of gantry point cloud

 Bottom edge of gantry mesh model

Bottom edge of gantry mesh model

The excellent free software tool, Cloudcompare displays the deltas of each dimension between the two points. For this experiment only the delta Z is of concern. Three sets of points were measured and tabled below.

Screen Shot 2018-03-12 at 9.48.08 am.png

What is striking is that the mean of the sets are surprisingly similar with approximately 10mm difference in accuracy. Although the laser scan standard deviation is double that of the photogrammetry, it can not be used for a rigorous determination due to the small data set of three points. The accuracy result of 9mm is less than the precision of the lidar set. This is a clear indication that the sample set is too small for detailed analysis. However, what is important is the similarity between the lidar and photogrammetry results.

So what is the accuracy of the SiteSee model? The answer is going to be very specific. Over 50m vertical,  SiteSee's achieves a variation of just +-10mm. Take the statement with a realisation that it is not a simple process to determine real world measurement accuracy.

What does this mean for tower owners? 
Maintaining accurate as-built documentation for tower owners is critical for a number of reasons.  Firstly, it ensures equipment has been installed as per design.  Current practices lack transparency - relying on subcontractors to install and audit their own work.  While it is common for 'spot checks' to be performed on a sample of new builds / upgrades, sometimes non-approved installations fall through the cracks - often only being picked up outside of the typical twelve month contractor liability period.  

By utilising SiteSee's solution whenever a physical change / upgrade / decommission occurs, SiteSee's infrastructure digital twin & automated as-built documentation helps tower infrastructure owners maximise tower utilisation.  The SiteSee solution also provides tower owners with new customer facing services to assist carriers & contractors to make more informed decisions throughout the planning and design phase without having to visit site.  

SiteSee Information Night

SiteSee was pleased to welcome guests from the 3D modelling and drone ecosystems to our Reality Modelling Information Night, at Brisbane startup hub River City Labs, on February 7. The evening was an opportunity for us to present the learnings from own reality modelling capture methodology, including recent the developments to our operations and technology.

With more than 100 registered attendees, the info night centered on bringing together some of the brightest minds in Australia’s UAV ecosystem and reality modelling community, with the intention of promoting collaboration towards delivering complete business solutions.

The night began with a project recap from Bentley Systems, presented by Account Manager Malcolm Doyle. Malcolm demonstrated the ongoing benefits to business achieved with a presentation of Bentley’s 2015 Philadelphia project, which saw more than 28,000 images combined into a virtual mesh of the city. Drawing from both aerial- and ground-captured data, they were able to create a high-fidelity reconstruction to assist in site management and security in preparation for a visit from the Pope. The Philadelphia project highlighted the real-world applications and use cases of reality modelling, and the functional impacts possible through clever utilization of digitized twins.

Spatial Consultant Marc Rietman then ran us through the fundamentals of photogrammetry, its benefits and limitations, and its importance to the reality modelling process. Marc explained how the photogrammetry process can be achieved with widely accessible hardware, as long as the data capture process is completed in a specific manner to ensure image quality and accuracy; his emphasis on the importance of efficient data collection methodology was later reinforced by SiteSee CEO Lucio Piccoli.

 

SiteSee Co-Founder Lachlan Crane continued the proceedings by detailing how SiteSee’s technical applications can benefit multiple industry verticals including telecommunications, natural resources, energy, civil and more.

Lachlan also highlighted the ways that our automated capabilities enable us to provide interactive, accurate reality models to deliver useful data for our clients. He handed over to Lucio, who explained the developments to our capture methodology from a more technical perspective.

Lucio detailed how SiteSee places an emphasis on delivering a technical service that is scalable, repeatable, accurate and commercially applicable. To ensure this, SiteSee’s data-capture methodology has been streamlined to leverage off-the-shelf equipment, step-by-step instructions for capturing imagery, and purpose-designed autopilot applications.

Lucio explained how we coordinate the entire capture process from flight path, camera specs, calibration, and ground control, with the intention of optimizing processing efficiency and minimizing our clients’ wait for the final product.

The evening concluded with a panel discussion featuring representatives from Nearmap, FlyFreely, Hovermap, Six Arms as well as a professional drone pilot – who collectively represent a sample of the services and technologies that are contributing to end-to-end solutions for enterprise clients. The goal of the panel was to identify and reflect on our businesses’ current approaches, and how we can work towards end-to-end reality modelling solutions collaboratively.

During the panel, professional drone pilot David Alam, from Australian Drones, described his experience on a recent data-capture course with SiteSee as an “eye-opener”. David emphasized that the course enhanced his understanding of the collaborative processes required to deliver a quality product that meets client needs.

Our panel concluded by addressing an array of questions from our eager audience members, with topics ranging from what to consider when deciding to implement a drone program, the potential of autonomous drones, effective data processing, and how to best manage operational challenges in the field.

Co-Founder & Sales Director Lachlan Crane said of the information night that it was “a tremendous success. We were blown away with the quality of local talent and technical capabilities coming out of Australia. There’s a lot of ‘noise’ about drone services and technical capabilities, which can hugely complicate matters for enterprise clients seeking to implement tailored business solutions. SiteSee firmly believes that best results are achieved through collaboration, and we will continue to support and collaborate with providers to deliver complete business solutions for our enterprise clients.”