A guide to better access, explore, and understand unstructured sensor data for autonomous driving development
To accelerate the speed of autonomous vehicle adoption, an increasing number of organizations and individuals are making their projects available to the public. Open data is fueling commercial and technological advancement in autonomous driving—one of most well known resources being the nuScenes dataset.
Developed by the team at Motional (formerly nuTonomy), nuScenes is one of the most popular open-source datasets for autonomous driving. The nuScenes dataset enables researchers to study a wide range of urban driving situations using data captured by the full sensor suite of a self-driving car. The first dataset of its kind, nuScenes was a key player in cultivating a culture of data sharing and collaboration within the mobility industry.
To further advance this mission, Motional recently partnered with Berlin-based startup SiaSearch to introduce a completely new way to interact with nuScenes—by using a data curation platform to delve deeper into the data than ever before.
This guide will walk you through how to better access, explore and understand the nuScenes dataset with the SiaSearch platform.
What is the nuScenes Dataset?
The nuScenes dataset is one of the largest public datasets for autonomous driving. The dataset contains a rich library of meticulously hand-annotated scenes collected by real self-driving cars. Recorded in Boston and Singapore, nuScenes features a diverse range of traffic situations, driving maneuvers, and unexpected behaviors.
The dataset includes:
- Full sensor suite: 32-beam LiDAR, 6 cameras and radars with complete 360° coverage
- 1000 urban street scenes, 20 seconds each
- 1,440,000 camera images
- 23 classes and 8 attributes
Accessing nuScenes data in SiaSearch
To access the data yourself, you’ll need to sign up for a free account on SiaSearch. After creating an account, you can immediately load and visualize nuScenes data within the web-based GUI.
Now that we’re all set up, let’s start exploring! Upon loading nuScenes in SiaSearch, you’ll see the main dashboard, which captures key features of the dataset. This view lets you quickly understand the overall dataset composition, as well as identify any gaps in data distribution.
Querying the nuScenes Dataset
Having a holistic view of the dataset, while useful, is not enough. The ability to drill into specific subsets can uncover insights and imbalances in the data—a critical step in model building and validation.
SiaSearch makes every piece of nuScenes data searchable against all available and auto-extracted dimensions using its intelligent search interface. The platform features two ways to search for the exact sequences you want, using either a visual or code interface. The visual query lets you select extractors from a list of semantic attributes and driving situations, while the code query functions like any API call statement would.
For example, if we are searching for rainy weather scenes with over three cars in view, you could submit the following query using the code interface:
vehicle_following = 'True' AND precip_type = 'RAIN' AND
road_joint = 'True' AND forward_velocity >= 5 AND num_cars >= 3
Within seconds, the API returns clips that match the query attributes. This makes it easy to search the entire data lake to return situations that fit the case in question. From the search interface, specific clips can be selected by clicking into them to reveal more details, such as charts for forward velocity and acceleration profiles.
Curating custom training datasets from nuScenes
Another key benefit of using the platform is the ability to transform data to produce custom training datasets. The flexibility of the SiaSearch platform allows you to finetune data for training and testing to see a measurable improvement in your models.
For example, you may want to isolate the data right before or after an unprotected left turn in order to contextualize vehicle behavior. The SiaSearch interface gives you the ability to adjust the length of snippets to match your requirements in just a few clicks.
Here are just a few ways you can use the SiaSearch to transform data:
- Re-adjust clip length by clicking on the adjust clip button in the replay viewer. Easily trim or expand snippets according to your needs.
- Add custom tags to selected segments to produce new subsets of data.
- Leave comments for collaborators on a frame or sequence level to notify them of any questions or feedback.
Now that we’ve selected our training samples, we need to find a way to feed the data back into the computer vision model.
Exporting data from SiaSearch
The final step is to export your data into a format you can import directly into your model. To access the export page, add one or more snippets to your export list from either the search or playback pages.
On the export page, you can review a breakdown of the selected snippets based on their associated queries, as well as basic statistics to get an idea of the quantity and contents of the export. When you’re happy with your results, you can export the raw data and or specific metadata attributes as a parquet file to feed back into your model or validation pipeline.
Carefully curated data is critical in the development of any great computer vision model. At SiaSearch, we are determined to accelerate ADAS development by empowering engineers with full access and visibility into their data.
If you found this article interesting, please leave a comment and follow us for similar content. For more information about the SiaSearch platform, please check out our website at https://www.siasearch.io/.
nuScenes is available for commercial use under our commercial license agreement. Please reach out to [email protected] to learn more.
Using data curation tools, engineers can get a better understanding of the data they’ve collected, identify the most important subsets and edge cases, and curate custom training datasets to feed back into their models.The role data curation tools play in machine learningThe best data curation tools enable you to:Visualize large scale data: Make it easy to obtain insights on key metrics, as well as the general distribution and diversity of your datasets regardless of sensor type and format.
Curate diverse scenarios: Identify the most interesting segments within your dataset, and manipulate them within the tool to create completely customized training sets.Seamlessly integrate: The tool should fit well within your existing workflows and toolset.What are the best data curation tools for computer vision?With an overwhelming amount of AI products and platforms popping up year after year, how do you know which will provide the most value?
Based on our experience, we are sharing our honest reviews of the top tools, hoping that this will be of use for engineers searching for a data curation solution.Read on below to find out which data curation tool is the best fit for your computer vision project.Aquarium LearningAquarium is a data management platform that aims to make it easy to identify labeling errors and model failures.
With Aquarium, users can version and combine model predictions with their ground truth.Aquarium is especially focused on curating and maintaining training datasets, catering less to raw data management use cases.
They also support multiple annotation types, such as classification, detection, and segmentation.Interactive model evaluation - Users can manipulate evaluation thresholds and obtain interactive visualizations to obtain required samples quickly.Collaborative features - Users can collaborate with each other on the Aquarium platform to build data subsets, associate them with issues, and identify new data for annotation.FiftyOneDeveloped by Voxel51, FiftyOne is an open-source tool to visualize and interpret computer vision datasets.
Today, the platform lacks collaborative features; for example, a single instance cannot host multiple user accounts.Key Features:Model & dataset zoo - FiftyOne taps into TF and Pytorch dataset zoos to provide access to a variety of open datasets and open-source models.Advanced data analysis - Via the Brain, a separate closed-source Python package, users can quantitatively assess the uniqueness, mistakenness, and hardness of data.External integrations - FiftyOne directly integrates with popular annotation tools such as LabelBox.
Especially amid movement restrictions induced by the COVID-19 pandemic, research shows that global online sales jumped to $26.7 trillion in 2020.
With the rise of ecommerce, one thing is abundantly clear: brick-and-mortar retailers need to innovate if they want to stay competitive.
Among the most promising applications of computer vision include inventory management, loss prevention, automated checkout, and behavioral analytics.
The company holds 23 patents on its technology and can analyze images from phones, in-store cameras, and grocery store robots.Trax uses computer vision technology to scan shelves in stores and identify what is neededAutomated checkoutStandard.ai: Previously known as Standard Cognition, Standard.ai’s automated checkout solution is made to fit with retailers’ existing stores and technology.
Standard doesn't use any facial recognition or biometrics, and all deployments are on-premise to ensure maximum performance and security for retailers and shoppers alike.Trigo: Using proprietary algorithms and affordable off-the-shelf sensor kits, Tel Aviv-based Trigo allows retailers to analyze anonymized shoppers’ movements and product choices in real time.
Once a customer reaches a certain threshold, the system sends an alert, along relevant video clips, to the appropriate staff member.Clips from the VaakEye product demo videoBehavioral analyticsDeep North: Deep North provides an analytics platform that builds real-time video intelligence for retailers based on video data from CCTV and other cameras that those retailers already use.
When these modes of transport carry so much material, it becomes crucial to monitor and track their movement across the globe.Satellite Image analysis provides a great solution to such problems.
These image analysis not only help in tracking the given subject but also helps in monitoring the surroundings of the target.
With ships detection, and combined with data of other ships routes and live status, (Data from other sources), many security features can be built.
Also, search and rescue operations can be further improved in terms of their response time and location accuracy.
These Applications require large quantities of data, and with the recent availability of satellite images from many tech giants and government organizations, it has become an emerging field in the aerospace domain.Rhammell on Kaggle provided a cleaned dataset of satellite imagery, separately for each of the categories, i.e.
Images were derived from PlanetScope full-frame visual scene products, which were ortho-rectified to 3-meter pixel size.
Read an overview about top deep learning frameworks 2020 and deep learning frameworks comparison of such platforms TensorFlow,Torch,Deeplearning4j,CNTK ,Keras,ONNX,MXNet and Caffe, How deep learning frameworks are play integral part in Artificial intelligence and Machine learning.Given that deep learning is the key to executing tasks of a higher level of sophistication, building and deploying them successfully proves to be quite the herculean challenge for data scientists and data engineers across the globe.
Today, we have a myriad of frameworks at our disposal that allows us to develop tools that can offer a better level of abstraction along with simplification of difficult programming challenges.
Machine learning in business helps in enhancing business scalability and improving business operations for companies across the globe.
But NLP technology is now advancing at a breathtaking rate, opening up new possibilities for conversational AI.
AI will augment the work of human clinicians in hospitals in various ways.As one example, Gauss Surgical uses computer vision to monitor blood loss during childbirth.
The vision of precision medicine is more ambitious, the technical challenges more complex, and the potential impact greater than perhaps any other application discussed here.In a nutshell, the field of precision medicine aspires to create treatments that are individualized for each patient based on his or her particular genetic, environmental and behavioral context.Precision medicine is not a new concept, but the advent of “big data” (especially genetic data) and modern machine learning have brought its full realization within reach.
Several trillion gigabytes of health data will be generated this year, a figure that would have been unimaginable a few short years ago.The premise of precision medicine is that if a computational system knows your entire genome, your metabolic profile, your microbiome composition, what foods you eat, how often you exercise, how much you sleep, and a thousand other data points about you; and it also understands a disease’s particular pathway in your body down to the molecular level; then it can synthesize all this information and craft a pharmaceutical and/or behavioral regimen specifically tailored to optimize your body's response.No human could ever perform such herculean data crunching and latent pattern recognition.
The company is focused on cancer treatment, although it has recently devoted resources to the fight against COVID-19.Other well-funded companies in this category include Syapse and GNS Healthcare.Precision medicine has for decades stood as a tantalizing but unfulfilled possibility.
AI can play a key role here.Provider OperationsEvery time a patient interacts with a healthcare provider, dozens of support processes take place in the background: patient check-in, benefit and verification discovery, claims processing, invoicing, prescription orders, supply chain management, and more.
