Detecting Functionality-Specific Vulnerabilities via Retrieving Individual Functionality-Equivalent APIs in Open-Source Repositories

Functionality-specific vulnerabilities are defined as vulnerabilities that mainly occur in APIs with specific functionalities. These vulnerabilities are often caused by a lack of crucial operations during implementation, such as input validation. Listing 1 is the patch of an example functionality-specific vulnerability, CVE-2023-27603, which affects the fileToUnzip API in a widely used third-party repository, Apache Linkis. This vulnerability corresponds to a representative type of functionality-specific vulnerability, CWE-22 (i.e., improper limitation of a pathname to a restricted directory). Lines 16-21 in Listing 1 of this patch explain how this vulnerability is exploited and fixed. Specifically, the vulnerable API fileToUnzip forgets to check whether the file paths (filePath in Line 16) in the input zip file (zipFilePath) contain improper relative paths, e.g., “..”, which has the risk of unzipping into a restricted directory (in Line 23).

Functionality-equivalent APIs refer to a collection of APIs that provide equivalent/similar functionalities, such as “unzip a zip file” or “parse a JSON file”. Take Listing 1 and Listing 2 as examples. These two APIs, which come from two repositories (Apache Linkis and Light4J), respectively, perform the same functionality: unzipping an input zip file to a specific path.

Based on the preceding definitions, functionality-equivalent APIs often share similar or even the same functionality-specific vulnerabilities. For example, the unzip API in Listing 2 is also vulnerable to a path traversal vulnerability, CVE-2023-33484, and is fixed by invoking the validateZipOut method before unzipping. However, the implementations of these APIs might diverge substantially. For example, fileToUnzip’s source code spans only $27$ lines while that of unzip spans $46$ lines.

Some existing approaches [43, 26, 36] are designed to detect recurring vulnerabilities, which come from reused code or shared code logic. For example, MVP [43] utilizes hash values based on the code property graphs of known vulnerable APIs to match similar APIs in open-source repositories. However, these approaches rely heavily on matching the preceding implementation-based features. This reliance may produce high false negatives in detecting functionality-specific vulnerabilities while the affected APIs have diverse implementations.

Some other approaches [41, 52, 39, 8, 38, 37, 42] learn code features extracted from vulnerable APIs/methods including existing vulnerabilities. For example, Devign [52] uses a Graph Neural Network (GNN) to learn the code property graphs (e.g., control/data-flow graphs) of vulnerable APIs. These approaches can effectively detect vulnerabilities caused by incorrect implementation, such as use-after-free, because the source code of their affected APIs has specific patterns, e.g., a pointer operation after a free API invocation. However, these approaches are ineffective in detecting functionality-specific vulnerabilities because their affected APIs have diverse implementations without a common pattern. Additionally, these existing approaches also face limitations in generating PoCs for the detected vulnerabilities. A recent study [46] shows that deep-learning models, including LLMs, struggle to generate inputs to cover a specific code branch, let alone generate inputs to exploit a specific vulnerability.

Our proposed approach has two main advantages when compared to existing work [29, 11, 6, 50, 47], which models the task of retrieving functionality-equivalent APIs as an API mapping task. First, existing approaches are less effective. Similar to vulnerability-detection approaches, existing API mapping approaches also overemphasize API bodies without being able to handle diverse implementations of functionality-equivalent APIs. Second, existing approaches are less generalizable because they are trained/fine-tuned on specific datasets, e.g., mapping first-party APIs in Java libraries to C# ones [29, 11, 6, 50] or Swift ones [47].

Algorithm 1 presents the pseudocode of APISS’ module of retrieving functionality-equivalent APIs. The algorithm takes as input a source API of an existing vulnerability ($srcAPI$), a set of candidate APIs ($candidateAPIs$), the weight of each feature (${𝒲}_i$), and the desired number of functionality-equivalent APIs ($N$). This algorithm outputs $N$ functionality-equivalent APIs associated with $srcAPI$.

In Lines 1–7, we compute each feature’s embedding of a given API. In Line 4, we compute the raw embedding of each API feature, ${ℰ}_i(API)$. Then in Line 5, we normalize each embedding vector, denoted as $\widehat{{ℰ}_i}(API)$. Our rationale for normalization is to ensure that the distances of each feature are within the same ranges.

In Lines 8–13, we compare the API embeddings of $srcAPI$ with those of $candidateAPIs$. In Lines 8–10, we generate the embeddings of both the $srcAPI$ and each candidate API using the getEmbeddings method. Then in Line 10, we define the distance of each pair of API embeddings as the weighted distance of each feature. Here, the distance of each feature is defined as the L2 norm [27, 12] of their embeddings’ subtraction, ${\Vert {ℰ}_i(src)-{ℰ}_i(dst)\Vert }_2$.

The performance of Algorithm 1 depends on the selection of feature weights (${𝒲}_i$). To determine the optimal set of values, we employ a global optimization technique. Specifically, we utilize the Basin-hopping algorithm [24], which has been widely applied in high-dimensional search problems [33, 48]. The optimization is performed on a dataset comprising Java and C# APIs (as shown in Table 6), and the resulting weights are subsequently used in the vulnerability detection process.

To enhance retrieval efficiency and mitigate the runtime costs of Algorithm 1, we utilize a vector database [20, 14]. Algorithm 1 requires calculating $|srcAPIs|$ * $|candidateAPIs|$ pairs of distances for all $|srcAPIs|$ as the distance between the $src$ API and each $dst$ API is computed in Lines 9-11. On the contrary, a vector database substantially reduces the runtime cost to $|srcAPIs|$ + $|candidateAPIs|$ by storing and retrieving the embeddings of $candidateAPIs$ while querying each $srcAPI$. During implementation, we choose Faiss⁷⁷7https://faiss.ai/index.html as our vector database and modify the selection criteria equivalently in Lines 11-12:

where ${ℰ}_i(API)$ is the i-th feature embedding of $API$, ${\widehat{ℰ}}_i(API)$ is the normalized embedding, and $\widehat{ℰ}(API)$ is the concatenated embeddings of all $API$’s features. Thus, we store only $\widehat{ℰ}(dst)$ of all candidate APIs in our vector database.

Our solution leverages our observation that a functionality-specific vulnerability’s PoC generally consists of two parts: input construction and API invocation. Input construction, i.e., creating specific objects or data structures, remains consistent across functionality-equivalent APIs. For example, in the PoCs of CVE-2023-27603 and CVE-2023-33484 (Listing 3), the construction of a zip file containing a malicious file is identical (Lines 12–30 in both PoCs). This zip file attempts to access a restricted path using the relative address “../../a/b/c/poc.txt”. On the contrary, API invocation, i.e., the actual invocation of the targeted API, varies minimally and comprises a smaller code portion of the PoC. Lines 7–8 demonstrate that the “unzip” method is called in the former CVE and “NioUtils.unzip” is called in the latter, both aiming to unzip the malicious zip file. Notably, the parameters for these invocations, including the path of the zip file and the targeted extraction path, remain identical/similar across different APIs.

The preceding observation highlights the feasibility of adapting PoCs between functionality-equivalent APIs with minimal adjustment. Thus, we design a semi-automatic schema to migrate the PoC of existing vulnerabilities with two steps. First, for each type of vulnerability, we collect a representative affected API with its PoC, and manually create a heuristic-based template for PoC migration. The details are listed in Table 1. Second, for the API invocation part, we migrate the usage of each source API into the destination API based on a simple parameter-matching algorithm [45]. As for those cases where automatic migration fails, we manually migrate them. Specifically, we look up the documents in their corresponding repositories or directly consult with search engines.

We employ API fuzzing to further exploit functionality-specific vulnerabilities in functionality-equivalent APIs whose source APIs are associated with specific vulnerability types. Our rationale is that identical inputs may not trigger the same or similar vulnerabilities in different APIs if their functionalities diverge slightly. Thus, APISS employs a fuzzer to generate more inputs to exploit potential vulnerabilities. We focus our fuzzing efforts on APIs whose inputs are amenable to mutation, as some constructed inputs cannot be effectively fuzzed. The vulnerabilities whose affected APIs require fuzzing are detailed in the “Need Fuzzing” column of Table 1.

We collect source vulnerabilities from a Java vulnerability dataset used by VulFixMiner [51], MiDas [30], and CompVPD [10], including 1,359 vulnerabilities from 127 Java repositories. For each vulnerability in this dataset, we search for its related issue from the commit message in its patch, and the reference links in its vulnerability reports in NVD⁸⁸8https://nvd.nist.gov. Then, for each issue with an executable PoC, we manually validate this PoC. For each type of functionality vulnerability, we choose the earliest PoC as the template one for migration (as shown in Table 2).

We extract 42,612 candidate APIs from 180 randomly sampled Java repositories on GitHub. Specifically, we adopt tree-sitter⁹⁹9https://tree-sitter.github.io/tree-sitter/, a widely-used parser, to extract the elements of each API, including doc string, signatures, and bodies.

We compare APISS with three state-of-the-art deep-learning models: GGNN [23] (used by Devign [52], Funded [38], and ReVeal [8]), GCN [21] (used by IVDetect [22]), and RGCN [9] (used by Vu1SPG [49]). Although these models are originally trained on C/C++ vulnerability datasets, we retrain them on the preceding Java vulnerability dataset to ensure a fair comparison in detecting Java vulnerabilities. We also compare APISS with another state-of-the-art recurring-vulnerability detection approach, MVP [43]. We exclude AHPH [26] and ReurScan [36] as they are specialized for detecting specific recurring vulnerabilities (non-functionality ones).

At the time of writing, APISS has detected 179 new vulnerabilities. As shown in Table 3, 60 have received new Common Vulnerabilities & Exposures Identifiers (CVE IDs), 18 have been confirmed by developers without CVE IDs, 87 are currently under review, and 14 have been reported by others. Vulnerabilities with CVE IDs are listed in Table 4. Among the 60 vulnerabilities with new CVE IDs, 15 are classified as high-risk and 3 as critical based on the CVSS Base Score [35]. This result highlights APISS’ effectiveness in detecting new vulnerabilities, bringing high value to security practice.

Table 3 shows that the majority of vulnerabilities detected by APISS belong to CWE-121 (Stack-based Buffer Overflow) and CWE-400/835 (Uncontrolled Resource Consumption/Infinite Loop). According to our manual investigation of the PoCs of these vulnerabilities, they are triggered by unexpected JSON inputs, especially those that are deeply nested. For example, Listing 5 shows the PoC of CVE-2023-34610, a new vulnerability (belonging to CWE-121) detected by APISS. Its input is constructed in Lines 3-22 and has the following format: “$\underset{9999}{[[\mathrm{\dots }[}$ 0 $\underset{9999}{]]\mathrm{\dots }]}$”.

Table 3 also shows the false-positive numbers of each type of vulnerabilities. APISS incorrectly identifies 263 false-positive vulnerable APIs, achieving a high precision score of 0.40. It is mainly because we discard functionality-equivalent APIS for which APISS fails to generate its PoC.

Listing 4 presents an example of a false-positive vulnerability. The issue arises due to the use of a subnet mask of /2, which results in an excessively large capacity (i.e., 1,073,741,822) when initializing an ArrayList. The maintainer has acknowledged that a validation check could be added to Ipv4Util to handle such illogical inputs and prevent errors.

However, the maintainer does not classify this case as a vulnerability, stating that “A subnet mask is typically used to divide an IP address range into smaller networks, so a /2 subnet is too large for any practical use, particularly for a public IP address such as 213.3.13.2.”

It is worth noting that false positives count for a small portion because most of them are crash-like vulnerabilities, which can be automatically verified. Such false positives rejected by repository maintainers account for a small fraction (i.e., less than 20%). Given the overall end-to-end effort (80 human-hours, as detailed in Section 5.1) and the effectiveness of our approach (identifying 179 new vulnerabilities and 60 new CVE IDs), the occurrence of such false positives remains within an acceptable range.

Table 5 illustrates the proportion of vulnerabilities detected by APISS and also detected by existing baselines. Overall, our baseline approaches can detect at most 58% of these vulnerabilities while MVP can detect 41% of them (corresponding to their recall values). Additionally, these baseline approaches encounter substantial false positives. MVP achieves a precision score of only 0.41 while the precision scores of existing deep-learning approaches (GCN, RGCN, and GGNN) are 0.02, 0.04, and 0.03, respectively. Such false positives will lead to a substantial manual costs when these identified vulnerabilities are validated. This result indicates that APISS is more effective in detecting functionality-specific vulnerabilities when compared with baselines. The main reason for their ineffectiveness is their overemphasis on API bodies disregarding the various implementations of functionality-equivalent APIs. On the contrary, APISS utilizes the similarities among API doc strings and signatures, thus successfully detecting these vulnerabilities.

We select two state-of-the-art API mapping approaches as our baselines, Api2Api [29] and SAR [6]. Both approaches employ a seeding step that is critical to their performance. For fair comparison, we evaluate all three seeding strategies (used in their paper): Random Seeds, k-Fold Seeds ($k=1,2,3,4$), and Signature-Based Seeds. For each strategy, we choose the optimal seed number that maximizes accuracy, based on the highest reported results in their paper [6].

We use the same dataset, Java2CSharp [18], as our baselines. This dataset maps Java First-Party APIs to functionality-equivalent C# First-Party APIs. It includes 1570 Java APIs, 2457 C# APIs, and 860 pairs of functionality-equivalent APIs.

Given that the API source code in Java2CSharp is not available and the SAR repository¹²¹²12https://github.com/bdqnghi/SAR_API_mapping does not provide complete details of API bodies and doc strings, we recollect all First-Party Java and C# APIs from jdk8u40¹³¹³13https://hg.openjdk.org/jdk8u/jdk8u40/jdk/file/c7bbaa04eaa8/src and .NET 4.8. for Windows January 2020 Update¹⁴¹⁴14https://referencesource.microsoft.com/DotNet48ZDP2.zip.

To keep the same setting with our baselines, we use Top-k Accuracy ($k=1,5$) as our evaluation metric. Top-k Accuracy measures the accuracy of API mappings by assessing whether the functionality-equivalent C# API for a given Java API appears within the top $k$ positions of the predicted API list.

Table 6 shows the Top-k Accuracy results of APISS and baselines. APISS substantially improves the API mapping accuracy with an increase of 22% in Top-1 Accuracy ($0.81-0.59$) and an increase of 7% in Top-5 Accuracy ($0.84-0.77$). This improvement is attributed to APISS’ use of LLMs, which effectively comprehend API functionalities.

We also observe that comparing the embeddings of only signatures achieves higher accuracy than our baselines, while comparing the embeddings of only doc strings or bodies achieves relatively lower accuracy. This result indicates that API signatures are the most critical feature for API mapping than doc strings or bodies.

The contribution of each feature in APISS’ embedding models, ChatGPT and CodeLLaMa, are depicted in Figure 2 and Figure 3. Each feature’s contribution percentage is noted within brackets, and the scatter points represent the feature’s contribution on each input API.

Both figures reveal that the three most important features are $full\mathrm{\_}sign$, $doc$, and $Full\mathrm{\_}DS$. These results highlight that both API doc strings and signatures (and their combination) mainly contribute to API mapping, while the API body has a negligible contribution (3% and 2%). Thus, the preceding results effectively support our proposal of utilizing API doc strings and signatures (instead of bodies) in API mapping.

Additionally, we notice two main differences from these figures. First, documentation strings show high value in smaller models in API mapping. For example, the most important feature for ChatGPT is $full\mathrm{\_}sign$ (API signatures) while that of CodeLLaMa is $full\mathrm{\_}ds$, the combination of API doc strings and signatures. Second, features that include full signatures, which include package and class names (e.g., com.networknt.utility.NioUtils), are more effective than those with raw signatures. This result suggests that this contextual information enhances LLM’s effectiveness in understanding API functionalities.

The end-to-end effectiveness of APISS highly depends on the expertise of our software engineers. Our evaluation, in which we have detected 179 new vulnerabilities, includes five software engineers. Each engineer has at least two years of experience in Java development and one year in fuzzing. The entire process of their manual costs takes approximately 50 man-days.

Figure 4 depicts the detailed workflow of how our engineers are involved in detecting new functionality-specific vulnerabilities. In this flowchart, steps requiring human intervention are highlighted in yellow. Adjacent to each step, we provide a breakdown of the average time costs.

The process of retrieving functionality-equivalent APIs, including extracting and validating the PoCs of existing vulnerabilities, requires approximately four hours per PoC. Our engineers read the vulnerability issue (if exists) and its URL links, and test the PoC within the affected software repository (on its vulnerable version). As shown in Section 5.3, a single validated PoC leads to the detection of an average of 17.9 new vulnerabilities, so the manual costs of four hours are affordable. Additionally, existing vulnerability databases, e.g., Snyk¹⁵¹⁵15https://snyk.io and VeraCode¹⁶¹⁶16https://www.veracode.com, have already collected a large number of PoCs although they do not open-source these PoCs to avoid potential risks.

The procedure of migrating PoCs and exploiting new vulnerabilities involves three steps requiring human intervention. First, our engineers identify the usage for each functionality-equivalent API that cannot be directly invoked by our parameter-matching algorithm [45]. This task applies to 20% of APIs, with each requiring 3-5 minutes to address. Second, for each API triggered by a potential vulnerability, our engineers verify whether the API inputs are legal. This step is necessary for 19.6% of APIs, also taking 3-5 minutes per API. Third, once a vulnerability is confirmed, reporting each one takes approximately 15 minutes. We exclude the manual costs of this step as the vulnerability is already detected and exploited. Therefore, the manual costs of migrating a PoC to a new vulnerability is within 10 minutes.

The runtime costs of APISS are shown in Table 7. For API mapping, APISS demonstrates high efficiency, with each step of embedding and retrieving taking less than one second. In the fuzzing process, we limit the maximum duration to two hours, determined based on the similarities between migrated inputs and potential vulnerability-triggering inputs. Consequently, the end-to-end fuzzing costs a total of 296 hours and successfully detects 144 vulnerabilities. These results confirm that the runtime costs of APISS are practical for real-world applications.

These approaches [43, 26, 36] are designed to detect recurring vulnerabilities, which come from reused code or shared code logic. For example, MVP [43] utilizes hash values based on the code property graphs of known vulnerable APIs to match similar APIs in open-source repositories.

However, these approaches rely heavily on matching the preceding implementation-based features. These approaches may produce high false negatives in detecting functionality-specific vulnerabilities as they rely heavily on matching the preceding implementation-based features while the affected APIs have diverse implementations. On the contrary, APISS considers functionality equivalence, a less strict constraint than implementation equivalence, to detect more vulnerabilities.

These approaches [41, 52, 39, 8, 38, 37, 42] determine whether a given method contains a potential vulnerability by learning code features of vulnerable APIs/methods of existing vulnerabilities. Devign [52] is a representative deep-learning approach. It uses Code Property Graphs (e.g., control/data-flow graphs) to represent the given method and designs a specialized Graph Neural Network (GNN) to determine whether this method is vulnerable.

These approaches can effectively detect vulnerabilities caused by incorrect implementation, such as use-after-free, because the source code of their affected APIs has specific patterns, e.g., a pointer operation after a free API invocation. However, deep-learning approaches are ineffective in detecting functionality-specific vulnerabilities because their affected APIs have diverse implementations without a common pattern, and they also face limitations in generating PoCs for detected vulnerabilities. On the contrary, APISS specifies the targeted vulnerabilities as functionality-specific ones, whose PoCs can be migrated from existing vulnerabilities.