Artifical Intelligence

SKYSCENES Dataset Could Lead to Safe, Reliable Autonomous Flying Vehicles

admin — Mon, 19 May 2025 13:31:53 +0000

SKYSCENES Dataset Could Lead to Safe, Reliable Autonomous Flying Vehicles admin Mon, 05/19/2025 - 09:31

Is it a building or a street? How tall is the building? Are there powerlines nearby?

These are details autonomous flying vehicles would need to know to function safely. However, few aerial image datasets exist that can adequately train the computer vision algorithms that would pilot these vehicles.

That’s why Georgia Tech researchers created a new benchmark dataset of computer-generated aerial images.

Judy Hoffman, an assistant professor in Georgia Tech’s School of Interactive Computing, worked with students in her lab to create SKYSCENES. The dataset contains over 33,000 aerial images of cities curated from a computer simulation program.

Hoffman said sufficient training datasets could unlock the potential of autonomous flying vehicles. Constructing those datasets is a challenge the computer vision research community has been working for years to overcome.

“You can’t crowdsource it the same way you would standard internet images,” Hoffman said. “Trying to collect it manually would be very slow and expensive — akin to what the self-driving industry is doing driving around vehicles, but now you’re talking about drones flying around.

“We must fix those problems to have models that work reliably and safely for flying vehicles.”

Many existing datasets aren’t annotated well enough for algorithms to distinguish objects in the image. For example, the algorithms may not recognize the surface of a building from the surface of a street.

Working with Hoffman, Ph.D. student Sahil Khose tried a new approach — constructing a synthetic image data set from a ground-view, open-source simulator known as CARLA.

CARLA was originally designed to provide ground-view simulation for self-driving vehicles. It creates an open-world virtual reality that allows users to drive around in computer-generated cities.

Khose and his collaborators adjusted CARLA’s interface to support aerial views that mimic views one might get from unmanned aerial vehicles (UAVs).

What's the Forecast?

The team also created new virtual scenarios to mimic the real world by accounting for changes in weather, times of day, various altitudes, and population per city. The algorithms will struggle to recognize the objects in the frame consistently unless those details are incorporated into the training data.

“CARLA’s flexibility offers a wide range of environmental configurations, and we take several important considerations into account while curating SKYSCENES images from CARLA,” Khose said. “Those include strategies for obtaining diverse synthetic data, embedding real-world irregularities, avoiding correlated images, addressing skewed class representations, and reproducing precise viewpoints.”

SKYSCENES is not the largest dataset of aerial images to be released, but a paper co-authored by Khose shows that it performs better than existing models.

Khose said models trained on this dataset exhibit strong generalization to real-world scenarios, and integration with real-world data enhances their performance. The dataset also controls variability, which is essential to perform various tasks.

“This dataset drives advancements in multi-view learning, domain adaptation, and multimodal approaches, with major implications for applications like urban planning, disaster response, and autonomous drone navigation,” Khose said. “We hope to bridge the gap for synthetic-to-real adaptation and generalization for aerial images.”

Seeing the Whole Picture

For algorithms, generalization is the ability to perform tasks based on new data that expands beyond the specific examples on which they were trained.

“If you have 200 images, and you train a model on those images, they’ll do well at recognizing what you want them to recognize in that closed-world initial setting,” Hoffman said. “But if we were to take aerial vehicles and fly them around cities at various times of the day or in other weather conditions, they would start to fail.”

That’s why Khose designed algorithms to enhance the quality of the curated images.

“These images are captured from 100 meters above ground, which means the objects appear small and are challenging to recognize,” he said. “We focused on developing algorithms specifically designed to address this.”

Those algorithms elevate the ability of ML models to recognize small objects, improving their performance in navigating new environments.

“Our annotations help the models capture a more comprehensive understanding of the entire scene — where the roads are, where the buildings are, and know they are buildings and not just an obstacle in the way,” Hoffman said. “It gives a richer set of information when planning a flight.

“To work safely, many autonomous flight plans might require a map given to them beforehand. If you have successful vision systems that understand exactly what the obstacles in the real world are, you could navigate in previously unseen environments.”

For more information about Georgia Tech Research at ECCV 2024, click here.

Summary sentence

Georgia Tech researchers created a new benchmark dataset of computer-generated aerial images that could allow autonomous flying vehicles to operate reliably and safely.

Summary

Few aerial image datasets exist that can adequately train the computer vision algorithms that would pilot autonomous flying vehicles. Judy Hoffman, an assistant professor in Georgia Tech’s School of Interactive Computing, worked with students in her lab to create SKYSCENES. The dataset contains over 33,000 aerial images of cities curated from a computer simulation program.

Dateline

Thu, 09/26/2024 - 12:00

Contact

Nathan Deen

Communications Officer

School of Interactive Computing

Associated importer

Keywords

go-ai

go-research

Artifical Intelligence

Center for Machine Learning

autonomous drones

drones; UAV; unmanned aerial vehicles

autonomous aircraft

computer vision

computer vision algorithm

computer vision systems

Georgia Tech Joins Virtual Reality Augmented Reality Association

admin — Tue, 29 Apr 2025 09:04:41 +0000

Georgia Tech Joins Virtual Reality Augmented Reality Association admin Tue, 04/29/2025 - 05:04

Georgia Tech is proud to announce its membership with the Virtual Reality Augmented Reality Association (VRARA) — marking a significant step in advancing innovation in the fields of augmented reality (AR) and virtual reality (VR). Founded in 2015, the VRARA is an international organization dedicated to fostering collaboration among industry leaders, promoting innovation, and advancing the adoption of immersive technologies. As a strategic sponsor of the VRARA’s Atlanta chapter, Georgia Tech plays a key role in advancing extended reality (XR) throughout the global ecosystem.

As a leading higher education institution, Georgia Tech has been at the forefront of AR and VR research and development. The Augmented Environments Lab has pioneered immersive digital experiences since 1998, focusing on applications that integrate physical environments with digital augmentation. Additionally, the Office of Information Technology (OIT) has been actively exploring XR technologies to enhance teaching, learning, and research, including partnerships with the Georgia Tech Library, the Center for 21st Century Universities, the Invention Studio @ Georgia Tech, and through efforts with faculty and student groups to explore immersive ER-based collaboration and co-design — the intersection of arts and XR.

“Joining the VRARA reflects Georgia Tech's commitment to driving innovation and supporting lifetime learners,” said Didier Contis, executive director of OIT’s Academic Research Technologies. “Our collaboration with VRARA will enable us to leverage our expertise in AR, VR, and artificial intelligence to develop innovative solutions that benefit our students, researchers, and the broader community."

Georgia Tech’s membership with the VRARA is also expected to bolster Atlanta's reputation as a hub for technological advancements and talent, aligning with the City of Atlanta’s dedication to attracting and retaining top professionals through initiatives that promote a diverse and skilled workforce. The collaboration is also poised to create new opportunities for research, development, and collaboration, further solidifying Atlanta’s position as a leader in the tech industry.

“Georgia Tech is one of the world’s premier universities,” said Adam Kornuth, president of the VRARA Atlanta chapter. “We couldn’t be more excited about our continued collaboration to drive strategic impact through innovation in the Southeast — and beyond.”

With over 50 chapters around the world, the VRARA boasts a network of more than 60,000 professionals and 4,000 organizations.

For more information about the Virtual Reality Augmented Reality Association, visit thevrara.com.

Summary sentence

Georgia Tech announces its membership with the Virtual Reality Augmented Reality Association (VRARA) marking a significant step in advancing innovation in the fields of augmented and virtual reality.

Summary

Georgia Tech announces its membership with the Virtual Reality Augmented Reality Association (VRARA) marking a significant step in advancing innovation in the fields of augmented and virtual reality.

Dateline

Tue, 04/29/2025 - 12:00

courtney.hill@gatech.edu

Keywords

oit feature

Artifical Intelligence

extended reality

News room topics

Science and Technology

Mercury ID

681893

Source updated

Tue, 04/29/2025 - 05:04

Minority English Dialects Vulnerable to Automatic Speech Recognition Inaccuracy

admin — Mon, 02 Dec 2024 16:41:41 +0000

Minority English Dialects Vulnerable to Automatic Speech Recognition Inaccuracy admin Mon, 12/02/2024 - 11:41

The Automatic Speech Recognition (ASR) models that power voice assistants like Amazon Alexa may have difficulty transcribing English speakers with minority dialects.

A study by Georgia Tech and Stanford researchers compared the transcribing performance of leading ASR models for people using Standard American English (SAE) and three minority dialects — African American Vernacular English (AAVE), Spanglish, and Chicano English.

Interactive Computing Ph.D. student Camille Harris is the lead author of a paper accepted into the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP) this week in Miami.

Harris recruited people who spoke each dialect and had them read from a Spotify podcast dataset, which includes podcast audio and metadata. Harris then used three ASR models — wav2vec 2.0, HUBERT, and Whisper — to transcribe the audio and compare their performances.

For each model, Harris found SAE transcription significantly outperformed each minority dialect. The models more accurately transcribed men who spoke SAE than women who spoke SAE. Members who spoke Spanglish and Chicano English had the least accurate transcriptions out of the test groups.

While the models transcribed SAE-speaking women less accurately than their male counterparts, that did not hold true across minority dialects. Minority men had the most inaccurate transcriptions of all demographics in the study.

“I think people would expect if women generally perform worse and minority dialects perform worse, then the combination of the two must also perform worse,” Harris said. “That’s not what we observed.

“Sometimes minority dialect women performed better than Standard American English. We found a consistent pattern that men of color, particularly Black and Latino men, could be at the highest risk for these performance errors.”

Addressing underrepresentation

Harris said the cause of that outcome starts with the training data used to build these models. Model performance reflected the underrepresentation of minority dialects in the data sets.

AAVE performed best under the Whisper model, which Harris said had the most inclusive training data of minority dialects.

Harris also looked at whether her findings mirrored existing systems of oppression. Black men have high incarceration rates and are one of the people groups most targeted by police. Harris said there could be a correlation between that and the low rate of Black men enrolled in universities, which leads to less representation in technology spaces.

“Minority men performing worse than minority women doesn’t necessarily mean minority men are more oppressed,” she said. “They may be less represented than minority women in computing and the professional sector that develops these AI systems.”

Harris also had to be cautious of a few variables among AAVE, including code-switching and various regional subdialects.

Harris noted in her study there were cases of code-switching to SAE. Speakers who code-switched performed better than speakers who did not.

Harris also tried to include different regional speakers.

“It’s interesting from a linguistic and history perspective if you look at migration patterns of Black folks — perhaps people moving from a southern state to a northern state over time creates different linguistic variations,” she said. “There are also generational variations in that older Black Americans may speak differently from younger folks. I think the variation was well represented in our data. We wanted to be sure to include that for robustness.”

TikTok barriers

Harris said she built her study on a paper she authored that examined user-design barriers and biases faced by Black content creators on TikTok. She presented that paper at the Association of Computing Machinery’s (ACM) 2023 Conference on Computer Supported Cooperative Works.

Those content creators depended on TikTok for a significant portion of their income. When providing captions for videos grew in popularity, those creators noticed the ASR tool built into the app inaccurately transcribed them. That forced the creators to manually input their captions, while SAE speakers could use the ASR feature to their benefit.

“Minority users of these technologies will have to be more aware and keep in mind that they’ll probably have to do a lot more customization because things won’t be tailored to them,” Harris said.

Harris said there are ways that designers of ASR tools could work toward being more inclusive of minority dialects, but cultural challenges could arise.

“It could be difficult to collect more minority speech data, and you have to consider consent with that,” she said. “Developers need to be more community-engaged to think about the implications of their models and whether it’s something the community would find helpful.”

Subtitle

Summary sentence

The Automatic Speech Recognition (ASR) models that power voice assistants like Amazon Alexa may have difficulty transcribing English speakers with minority dialects.

Summary

Interactive Computing Ph.D. student Camille Harris is the lead author of a paper accepted into the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP) this week in Miami.

Dateline

Fri, 11/15/2024 - 12:00

ndeen6@gatech.edu

Contact

Nathan Deen

Communications Officer

School of Interactive Computing

Associated importer

Keywords

speech recognition

bias

Research Horizons

go-research

go-researchnews

go-ai

Artifical Intelligence

inequality

News room topics

Society and Culture

AI4OPT and PSR Partner to Transform US Energy Industry

admin — Mon, 22 Jul 2024 22:44:49 +0000

AI4OPT and PSR Partner to Transform US Energy Industry admin Mon, 07/22/2024 - 18:44

This partnership in the advancement of AI and mathematical optimization to address pressing energy transformations in Latin America and the U.S. has formed between the NSF Artificial Intelligence (AI) Research Institute for Advances in Optimization (AI4OPT) at Georgia Tech and PSR, Inc. - Energy Consulting and Analytics.

PSR is a global leader in analytical solutions for the energy sector, providing innovative technical consultancy services and state-of-the-art power systems planning software. Their tools are used for detailed modeling of entire countries or regions and are utilized in over 70 countries. Together with AI4OPT, they aim to leverage advancements in AI and mathematical optimization to address pressing energy transformations in Latin America and the U.S.

Latin America boasts abundant renewable energy resources, especially hydropower, leading to one of the largest shares of renewables in its energy mix. However, expanding renewable energy capacity in Latin America and the U.S. to meet decarbonization goals will require system operational advances and new technologies that can adapt to current needs.

One focus of this collaboration will be studying how to efficiently incorporate pumped storage into the resource mix as a solution for long-duration storage. These plants act as large batteries, pumping water to higher reservoirs during low demand periods and generating electricity during high demand with minimal energy loss over time. This technology supports both short-term and long-term energy storage, making it crucial for managing the variability of intermittent renewables like solar and wind.

The complex and large-scale nature of the expansion problem, exacerbated by inherent uncertainty and time-coupled decisions, traditionally requires sophisticated optimization techniques. AI innovations now provide faster solutions and better representations of non-linear dynamics, leading to more cost-effective operations and optimized energy mix selection for the energy transition.

This collaboration plans to use machine learning to enhance power system operators' ability to perform faster security checks and screenings. As renewable energy sources introduce more variability, traditional methods struggle with the increasing number of scenarios needed for grid stability. Machine learning offers a solution by expediting these analyses, supporting the integration of more renewable energy into the system.

About PSR

PSR is a global provider of analytical solutions for the energy sector, spanning from insightful and innovative technical consultancy services to state-of-the-art specialized power systems planning software applied to the detailed modelling of entire countries or regions. Having its products applied in over 70 countries, PSR contributes to the research and development of optimization and data analytics’ tools for guaranteeing a reliable and least-cost operation of power systems, helping the countries achieve their decarbonization targets.

About AI4OPT

The Artificial Intelligence (AI) Research Institute for Advances in Optimization, or AI4OPT, aims to deliver a paradigm shift in automated decision-making at massive scales by fusing AI and Mathematical Optimization (MO) to achieve breakthroughs that neither field can achieve independently. The Institute is driven by societal challenges in energy, logistics and supply chains, resilience and sustainability, and circuit design and control. To address the widening gap in job opportunities, the Institute delivers an innovative longitudinal education and workforce development program.

Summary sentence

AI4OPT and PSR partner to transform the energy sector, focusing on renewable energy integration and long-duration storage solutions.

Summary

AI4OPT and PSR partner to transform the energy sector, focusing on renewable energy integration and long-duration storage solutions.

Dateline

Mon, 07/22/2024 - 12:00

breon@gatech.edu

Contact

Breon Martin

AI Research Communications Manager

Georgia Tech

Associated importer

Keywords

go-ai

go-researchnews

AI4OPT

Artifical Intelligence

AI4OPT, Artifical Intelligence, Optimization

News room topics

Campus and Community

Mercury ID

675562

Source updated

Mon, 07/22/2024 - 18:44

AI Hub Planned for CAU Under $2.79M NSF Grant with AI4OPT at Georgia Tech

admin — Mon, 22 Jul 2024 22:20:40 +0000

AI Hub Planned for CAU Under $2.79M NSF Grant with AI4OPT at Georgia Tech admin Mon, 07/22/2024 - 18:20

Clark Atlanta University (CAU), in collaboration with Georgia Tech’s NSF Artificial Intelligence (AI) Research Institute for Advances in Optimization (AI4OPT), has been awarded a four-year $2.79 million grant (Award ID 2402493) by the National Science Foundation (NSF) to create an AI Hub. This collaborative effort aims to advance AI education and research at minority-serving institutions, particularly historically Black colleges and universities (HBCUs).

This initiative, part of the NSF ExpandAI program, aims to boost minority-serving institution participation in AI research, education, and workforce development through capacity-building projects and partnerships within the NSF-led National AI Research Institutes ecosystem.

Building an AI community is no easy feat, but the CAU-GT/AI4OPT collaboration is prepared to meet it. The project, known as AIHUB@CAU, will be led by principal investigator Charles B. Pierre, associate professor in CAU’s Department of Mathematical Sciences.

"The mission of the grant aligns with the AI4OPT Faculty Training Program, which focuses on strategies to increase minority participation in AI research programs from HBCUs to other minority-serving institutions," said Pierre, who also leads the Educational and Diversity Initiatives at AI4OPT. "Our goal is to ensure diverse representation in the AI field."

The collaboration will use existing educational resources and infrastructure to build centers of excellence in AI and a community of empowered Black AI researchers.

"We anticipate challenges in developing coursework, including finding qualified industry professionals to teach and preparing academic professors unfamiliar with AI," Pierre said. "Our aim is to establish Ph.D. programs at CAU and position the university as a hub for AI training, addressing these issues head-on."

AIHUB@CAU will integrate industry partnerships to accelerate curriculum development and real-world applications. It expands AI education beyond machine learning to encompass decision-making and applications in fields like business analytics, cyber-physical security, and operations research.

Partially funded through NSF's Louis Stokes Alliances for Minority Participation program, this award underscores NSF's commitment to diversity in STEM fields through impactful educational and research initiatives.

"Establishing programs at institutions like Clark Atlanta University and AI4OPT at Georgia Tech provides students with essential resources and tools to succeed in this ever-evolving field," Pierre noted.

Goals and Structure of the AI Education Program

Main Goals of Creating AI Courses at the Undergraduate and Graduate Levels:

Close the gap of AI graduates from HBCUs at undergraduate and graduate levels.
Prepare HBCU students for the AI workforce.
Align with the vision of AI4OPT at Georgia Tech to "democratize access to AI education."

Impact on Students' Career Prospects and the AI Research Community:

Undergraduate courses and programs will prepare students for entry-level positions in the field.
Graduate courses and programs will prepare students for research and participation in the AI research community.

Role and Contribution:

AI4OPT at Georgia Tech will assist CAU with the development of both undergraduate and graduate courses and programs.
Offer research opportunities to CAU students at the undergraduate and graduate levels.
AI4OPT at Georgia Tech will be a partner in the established AI Research Hub.

Support for Development of MS and Ph.D. Courses:

Use current courses at Georgia Tech as a template.
Use the courses offered through the Faculty Training Program (FTP) of AI4OPT.

Foundational AI Courses:

Courses already taught by CAU faculty in the AI4OPT FTP.
Courses available at Georgia Tech.
New courses to be developed by AIHUB@CAU based on Intel material, focusing on computer vision and natural language processing.
Courses in applied optimization developed by AI4OPT.
New use-inspired AI courses teaching applications of AI in various domains, such as supply chains, security, chemistry, and manufacturing.

Research Opportunities:

The Undergraduate Research Program (URP) will provide students with early exposure to AI research, including summer internships at Georgia Tech and other AI4OPT sites.
The graduate programs will include an 18-month non-thesis master's degree with a summer internship and capstone project, and a two-year thesis master's degree supported by a six-month research project.

Structure of the New Master in AI Program:

Courses in five categories to support the master’s program:
1. Existing courses at CAU taught in the AI4OPT FTP.
2. Courses available at Georgia Tech.
3. New courses based on Intel material.
4. Applied optimization courses developed by AI4OPT.
5. New courses developed by AIHUB@CAU focusing on AI applications in various domains.

Collaborations and Internships:

Joint supervision of research projects by CAU and AI4OPT faculty.
Summer internships starting in 2026.
Capstone projects facilitated by Georgia Tech and industrial partners.

About AI4OPT

About Georgia Tech

The Georgia Institute of Technology, or Georgia Tech, is a top 10 public research university developing leaders who advance technology and improve the human condition. The Institute offers business, computing, design, engineering, liberal arts, and sciences degrees. Its nearly 40,000 students, representing 50 states and 149 countries, study at the main campus in Atlanta, at international campuses, and through distance and online learning. As a leading technological university, Georgia Tech is an engine of economic development for Georgia, the Southeast, and the nation, conducting more than $1 billion in research annually for government, industry, and society.

About Clark Atlanta University

Clark Atlanta University was formed with the consolidation of Atlanta University and Clark College, both of which hold unique places in the annals of African American history. Atlanta University, established in 1865 by the American Missionary Association, was the nation’s first institution to award graduate degrees to African Americans. CAU is also the largest of the 37-member UNCF institutions. CAU, established four years later in 1869, was the nation’s first four-year liberal arts college to serve a primarily African American student population. Today, with over 4,000 students, CAU is the largest of the four institutions (CAU, Morehouse College, Spelman College, and Morehouse School of Medicine) that comprise the Atlanta University Center Consortium.

About National Science Foundation

The U.S. National Science Foundation propels the nation forward by advancing fundamental research in all fields of science and engineering. NSF supports research and people by providing facilities, instruments and funding to support their ingenuity and sustain the U.S. as a global leader in research and innovation. With a fiscal year 2023 budget of $9.5 billion, NSF funds reach all 50 states through grants to nearly 2,000 colleges, universities and institutions. Each year, NSF receives more than 40,000 competitive proposals and makes about 11,000 new awards. Those awards include support for cooperative research with industry, Arctic and Antarctic research and operations, and U.S. participation in international scientific efforts.

Summary sentence

Clark Atlanta University, with AI4OPT at Georgia Tech, received a $2.79 million NSF grant to advance AI education and research at HBCUs, led by Charles B. Pierre.

Summary

Clark Atlanta University has received a $2.79 million NSF grant to establish an AI Hub in collaboration with the AI4OPT Institute at Georgia Tech. Led by Charles B. Pierre, the initiative aims to advance AI education and research at HBCUs, addressing underrepresentation in the field.

Dateline

Mon, 07/22/2024 - 12:00

breon@gatech.edu

Contact

Breon Martin

AI Research Communications Manager

Georgia Tech

Associated importer

Keywords

go-ai

go-researchnews

AI-CARING

AI4OPT

Artifical Intelligence

AI4OPT, Artifical Intelligence, Optimization

News room topics

Science and Technology

Artifical Intelligence

SKYSCENES Dataset Could Lead to Safe, Reliable Autonomous Flying Vehicles

What's the Forecast?

Seeing the Whole Picture

Keywords

Categories

Georgia Tech Joins Virtual Reality Augmented Reality Association

Related links

Keywords

News room topics

Minority English Dialects Vulnerable to Automatic Speech Recognition Inaccuracy

Addressing underrepresentation

TikTok barriers

Keywords

News room topics

Categories

AI4OPT and PSR Partner to Transform US Energy Industry

Keywords

News room topics

AI Hub Planned for CAU Under $2.79M NSF Grant with AI4OPT at Georgia Tech

Keywords

News room topics

Categories