Introduction

In the Artificial Intelligence ("AI") space, there is a steady rise in the popularity of tools such as ChatGPT. Under the hood, these chatbots leverage various AI tools. One such tool for extracting text or identifying objects from images is popularly known as an AI image recognition tool.

But the main question is, how secure are these tools? Certus Cybersecurity found two ways of adding malicious information within AI images: (1) by adding malicious data into the learning set and (2) by creating user input that can bypass how the data is processed.

To add malicious data into the learning set, technically, developers are the only ones who can infect it. However, this article will explore how external adversaries can bypass and process the data. We extensively researched scaling attacks and will discuss various abuse cases observed.

What is an AI Scaling Attack?

As described by Erwin Quiring, the general idea for scaling attacks is hiding an image inside another such that a new image is produced when the attack image is scaled down. According to their research, “several scaling algorithms are implemented with fixed-size convolution kernels ... [which cause] these algorithms to become vulnerable” (Quiring p.1369). To break this notion down, a kernel is a set of values that will be applied to groupings of pixels on the image. A convolution kernel is a mathematical function to process images, such as adding a filter, scaling, or blurring an image. Most popular libraries, such as OpenCV, TensorFlow, and Pillow, use the NearestNeighbor Scaling algorithm by default. The Nearest-Neighbor Scaling algorithm is a common and economic scaling convolution kernel, which is unfortunately vulnerable to scaling attacks. These libraries allow the configuration of different algorithms, which can assist in optimization and defense against malicious attack images.

For example, we can take two different images and reduce one image till it looks something different.

Abuse Cases of Scaling Techniques

Hiding Information

To understand the severity of scaling attacks, a security professional should review examples of how an adversary could utilize them. The most straightforward strategy would be for the attacker to hide information from AI. For example, on social media, a malicious user may upload a photo that would seem normal to the average person, such as a photo of a pet or family. However, if another adversary were to copy the image and scale it down using a vulnerable library and algorithm, the real image would be produced. This second action could expose sensitive information or a secret to pass to another attacker without being noticed.

Bypassing Safety Filters

Social media sites tend to have safety policies to stop uploading images that may be deemed too graphic or sensitive to the general public, e.g., images depicting violence or nudity. Typically, these images are processed through an AI-based safety filter after scaling the image to an efficient size. However, since AI scans images after scaling, threat actors could use a scaling attack to upload graphic content, bypassing safety measures. A good example is profile pictures in a messaging application where photos tend to be scaled down for friends lists. Below, a flower is hidden within the model tank to make the tank appear benign to the AI. When successful, a scaling attack would show the tank in the main profile but display a flower in the mini version of their profile picture instead.

Manipulating Learning Sets

A more complex example of a scaling attack would be to inject malicious images into an AI’s learning set. The learning set is the foundation for how an AI creates associations between images. An example from Quiring’s website, Image-Scaling Attacks and Defenses, is an AI set up to identify hazards and street signs to improve the safety of self- driving cars. If a malicious user can manipulate the learning set, they could inject an image similar to the attack image (below). For example, the AI may begin to associate the green square embedded in the stop sign with stopping. Future attempts from AI learning to identify street signs may compare a correct image with the now-incorrect original target image. Without intervention or prevention, this would quickly become problematic as self- driven cars could identify every ambiguous green square to mean STOP!

Mitigation

Quiring provides two ways to protect scaling functions from malicious images. One way is to implement libraries and algorithms that scale using a “dynamic kernel width” (Quiring p.1369). According to their research, using an Area-Scaling algorithm or a library that supports a dynamic kernel, such as Pillow, is a mitigation strategy, but the Nearest-Neighbor Scaling algorithm cannot be used in the same way. A second way for mitigation is to perform “Image Reconstruction”, – where the pre-scaled image is processed using a filter that attempts to manipulate where the pixels reconstruct after scaling. Specifically, Quiring proves that using a “Median Filter” and a “Random Filter” effectively ensures the pixels return to their intended position in the image. To help perform filtering and generate adversarial examples, Quiring created the 2019–scalingattack repository, which provides python scripts to describe the experiments in their research.

Proof of Exploit Repository

Building upon the original 2019-scaling attack repository, our team at Certus Cybersecurity created some updates to the repository to make it easier to use for security analysts, which can be found here.

The updated repo provides the following:

• Updated libraries to use Python 3.10
• Command line arguments
• Allows the use of custom images to create scaling attack examples
• Specifies a specific algorithm and library to use when generating an image
• Optionally, a percentage of scaling can be provided to make the attack scaling image clearer when used. An output folder of examples from each library, algorithm, and percent of scaling increments of 20 are also provided.

When creating an image and using the Area-Scaling algorithm or the Pillow library (excluding the Nearest-Neighbor algorithm), the script cannot generate an image since it is protected against such scaling attacks.

Key Takeaways

Many of these scaling attacks go beyond the scope of this post, but here are some key takeaways:

• Validate user input to ensure the data meets all criteria for the use case.
• Research how to use third-party methods safely when introducing user data.
• Use known and safe examples of malicious user input when testing an application.
• Use an Area algorithm or the Pillow library (not set to the nearest) for Image Scaling to help prevent attacks.
• Perform a filter on images the user provides to prevent image scaling attacks.

Conclusion

Before implementing a new technology into a company’s ecosystem, reviewing any security research on the subject is advisable. This brings security leaders up to speed on what to look out for and what questions to ask during the setup phase. For example, when implementing AI image scaling modules, an attacker may introduce malicious images which transform when scaled down. Therefore, choosing the correct libraries and algorithms is crucial in mitigating this attack.

The Area Scaling algorithm provides a dynamic kernel width that scales the image without combining the attack image’s pixels with the original image. Another defense is to create a filter on the image before scaling. The filter takes the median of nearby pixels to remove the attack image’s pixels. Once defenses are in place, look for test examples of what an adversary could send into the system.

After examples are collected, test the cases where a threat actor could interact with the system. For example, sending an image in a social media post or the actor can bypass security and add their images to the learning set of an AI. Considering these scenarios throughout the Software Development Life Cycle (SDLC) lowers the security risks for scaling attacks. In addition, security teams can become better prepared against threats by developing their own tests and mitigation strategies.

About the Author

Nickalas Porsch, Security Engineer, contributed to this publication.