Fluff

Master Thesis

Evaluating the Quality and Developer Modifications of ChatGPT-Generated Code Reused in GitHub Repositories

by

Full-Name1, Full-Name2 (VUNETID-1, VUNETID-2)

First supervisor: Name and surname Daily supervisor: Name and surname

Second reader: Name and surname

September 2, 2025

Submitted in partial fulfillment of the requirements for the VU degree of Master of Science in Information Sciences

DECLARATION OF AUTHORSHIP Use one declaration for each author.

I, [NAME], declare that this thesis titled, [TITLE] and the work presented in it are my own and that of my co-author. I confirm that: • This work was done wholly or mainly while in candidature for a research degree at this University. • Where any part of this thesis has previously been submitted for a degree or any other qualification at this University or any other institution, this has been clearly stated.

• Where I have consulted the published work of others, this is always clearly attributed. • Where I have quoted from the work of others, the source is always given. With the exception of such quotations, this thesis is entirely my own work.

• I have acknowledged all main sources of help. • Where the thesis is based on work done by myself jointly with others, I have made clear exactly what was done by others and what I have contributed myself.

Signed:

Date:

I, [NAME], declare that this thesis titled, [TITLE] and the work presented in it are my own and that of my co-author. I confirm that: • This work was done wholly or mainly while in candidature for a research degree at this University. • Where any part of this thesis has previously been submitted for a degree or any other qualification at this University or any other institution, this has been clearly stated.

• Where I have consulted the published work of others, this is always clearly attributed. • Where I have quoted from the work of others, the source is always given. With the exception of such quotations, this thesis is entirely my own work.

• I have acknowledged all main sources of help. • Where the thesis is based on work done by myself jointly with others, I have made clear exactly what was done by others and what I have contributed myself.

Signed:

Date:

2

Evaluating theQuality and Developer Modifications of ChatGPT-Generated Code Reused in GitHub Repositories

Full-Name1, Full-Name2 Vrije Universiteit Amsterdam Amsterdam, The Netherlands {name1|name2}@student.vu.nl

ABSTRACT Context. Large language models (LLMs), such as ChatGPT, are increasingly used to generate source code from natural language prompts. While these tools accelerate prototyping and reduce repet- itive coding tasks, their real-world adoption raises concerns about code quality, maintainability, and security.

Goal. This thesis investigates how ChatGPT-generated code is reused in public GitHub repositories, evaluates its quality using static analysis tools, and examines how developers modify such code after integration.

Method. We use the DevGPT dataset, which collects publicly shared ChatGPT conversations linked to GitHub repositories. From this dataset, we extracted Python and JavaScript snippets, validated their syntax, and analyzed them with linters such as Pylint, ESLint, and Bandit. We also tracked modifications in commit histories to identify how developers address or introduce quality issues over time.

Results. Only 11.6% of the 808 tracked snippets were reused in repositories, mainly Python. Linter analysis revealed that reused snippets contained an average of 18 issues, mostly related to style and maintainability. While developers fixed many of these issues, they also introduced new ones, resulting in a net decline in quality for most snippets.

Conclusions. ChatGPT-generated code can accelerate develop- ment but often requires significant human intervention to meet project standards. Developers tend to prioritize functional integra- tion over strict quality compliance, highlighting the need for AI tools that better align with coding conventions and best practices.

Keywords: ChatGPT, AI-generated code, code reuse, code qual- ity, static analysis, GitHub, software development, linters, Python, JavaScript, code modification.

1 INTRODUCTION Artificial intelligence (AI) has significantly transformed various sectors, and software development is no exception. A notable ad- vancement is the emergence of large language models (LLMs), such as ChatGPT by OpenAI [10], which can generate coherent natural- language text and executable code snippets in popular programming languages like Python and JavaScript. These capabilities enable de- velopers to automate routine coding tasks, accelerate prototyping, and even assist with debugging. However, despite these benefits, the increasing use of AI-generated code in real-world projects raises important concerns about its quality, reliability, and suitability for production environments.

To investigate these issues, this thesis relies on the DevGPT dataset [20], a curated collection of developer interactions with ChatGPT, including generated code snippets and metadata linked

to public repositories on platforms like GitHub. Code quality is a critical aspect of software engineering. Poor-quality code can introduce bugs, create security vulnerabilities, and increase long- term maintenance costs, which may offset the productivity gains achieved through AI assistance [9]. A systematic examination of ChatGPT-generated code is therefore necessary to understand its strengths, limitations, and practical usability in collaborative soft- ware development.

Although models like ChatGPT often produce syntactically cor- rect code, they lack deeper contextual understanding. As a result, AI-generated code can include logical errors, inefficient solutions, or deviate from established coding practices [24]. Such shortcomings are especially relevant in open-source and collaborative environ- ments, where code quality directly impacts trust, maintainability, and overall project success. By empirically analyzing the reuse and modification of ChatGPT-generated code, this thesis aims to provide an evidence-based understanding of how AI-generated code inte- grates into real-world projects and how it aligns with established software engineering practices.

1.1 Motivation The primary motivation for this research is the increasing reliance on AI-based tools in software development. As more developers integrate ChatGPT into their projects—whether to generate boil- erplate code, test algorithms, or assist in debugging—the need to rigorously evaluate the quality of its output has become critical. While ChatGPT can produce working code rapidly, it often lacks domain-specific reasoning and may fail to meet long-term require- ments [24].

This concern aligns with an industry-wide emphasis on improv- ing software quality. Static analysis provides a systematic way to evaluate AI-generated code, uncover recurring issues, and guide targeted improvements [9]. Applying these tools to AI-generated code provides an opportunity to systematically assess its quality, uncover recurring issues, and propose targeted improvements. By analyzing the DevGPT dataset in this context, this thesis bridges the gap between AI-generated code and established software en- gineering practices, offering insights that are directly applicable to both developers and researchers aiming to improve AI-assisted development workflows.

Finally, the results of this study have broader implications beyond individual developers. As AI-generated code increasingly appears in public repositories, understanding its quality and adoption patterns can influence best practices, guide the development of new tools, and inform educational approaches in programming. By providing empirical evidence rather than speculation, this work contributes

1

to a more grounded understanding of the role of AI in modern software engineering.

1.2 Research Questions This thesis is guided by three research questions that investigate the quality and real-world behavior of ChatGPT-generated code:

(1) What quality issues are most commonly identified by linters in ChatGPT-generated code snippets? This question categorizes and quantifies the most frequent deficiencies in AI-generated code, such as stylistic inconsis- tencies, potential bugs, code smells, or security risks. Identi- fying these patterns clarifies the model’s intrinsic limitations and highlights areas for improvement.

(2) How does reused ChatGPT-generated code on GitHub perform in terms of quality, as measured by linter- detected issues? This question evaluates the quality of reused AI-generated code in public repositories, revealing whether developer se- lection leads to improved integration quality or whether issues are propagated unchanged.

(3) How do developers modify reused ChatGPT-generated code on GitHub over time? This question analyzes how reused snippets evolve by com- paring their initial and final versions across commit history, showing whether developers fix existing issues, introduce new ones, or leave problems unresolved.

1.3 Solution To address RQ1, Python and JavaScript snippets from the DevGPT dataset were analyzed using static code analysis to identify common quality issues. This established a baseline view of how ChatGPT- generated code performs in terms of style, maintainability, and potential functional or security problems.

RQ2 examined the same snippets after their reuse in public GitHub repositories to determine whether developers improved or left these issues unaddressed before integration.

RQ3 compared the earliest and latest versions of multi-commit snippets to track how quality evolved over time, identifying which issues were fixed, which persisted, and which new ones were intro- duced.

This combined approach, evaluating both the intrinsic quality of AI-generated code and its real-world reuse, offers a more detailed view of ChatGPT’s impact, highlighting its technical limitations as well as how developers adapt it in practice.

1.4 Contributions This thesis has the following key contributions to the field:

(1) Comprehensive Quality Analysis: It provides a detailed assessment of ChatGPT-generated code quality, highlight- ing the most common linter-detectable issues in Python and JavaScript snippets. This analysis clarifies the model’s strengths and limitations, contributing to the ongoing dis- cussion about the role of AI in software development.

(2) Insights into Developer Practices: It examines how devel- opers reuse and modify AI-generated code in open-source

repositories, revealing the challenges and strategies involved in integrating such code into real-world projects.

(3) Recommendations for Improvement: Based on the find- ings, the thesis offers practical guidance for developers (e.g., reviewing and refining AI-generated code) and suggestions for AI researchers (e.g., improving models to reduce recur- ring issues). These recommendations aim to strengthen the collaboration between human expertise and AI assistance in software engineering.

2 BACKGROUND Large language models (LLMs) have enabled automated generation of code from natural language. Tools such as ChatGPT, trained on large code repositories, can produce Python or JavaScript snippets in response to free-form prompts [10]. Early evaluations focused on basic correctness, such as syntax and execution. However, real- world use in active projects involves more than basic correctness. Developers must ensure that generated code integrates with ex- isting systems, follows project-specific conventions, and remains maintainable over time.

GitHub, hosting hundreds of millions of public repositories, pro- vides an opportunity to observe how AI-generated code is reused. Developers often copy ChatGPT outputs into active codebases, but it is unclear how often these snippets persist, how they are received by automated quality tools, or how maintainers modify them over time [20]. Studying this reuse highlights the benefits of rapid gen- eration and the challenges of long-term integration.

In Python, Pylint and Flake8 enforce PEP 8 style guidelines and detect potential bugs. In JavaScript, ESLint and security-focused plugins identify unsafe or inconsistent patterns [9]. Applying these tools to ChatGPT-generated code reveals stylistic issues, missing dependencies, and potential vulnerabilities [16, 22], and compar- ing results between initial and later commits shows which issues developers fix or tolerate over time.

This study combines three elements: AI-based code generation, GitHub as a dataset of real-world reuse, and linter-based quality assessment. It addresses three research questions: (RQ1) how preva- lent is the reuse of ChatGPT-generated snippets on GitHub, (RQ2) what quality issues do linters detect in these reused snippets, and (RQ3) how do developers modify these snippets after reuse. These perspectives connect controlled evaluations of AI-generated code with its actual use and evolution in collaborative projects.

3 RESEARCH DESIGN In this study, we address three research questions aimed at under- standing the integration and impact of ChatGPT-generated code within software development projects.

All scripts, linter configurations, and analysis data used in this study are available for reproducibility at: https://github.com/NutKing/ GPT-thesis.

3.1 DevGPT Dataset The foundation of this research is the DevGPT dataset [17, 25], a curated collection designed to study developer interactions with ChatGPT. It leverages OpenAI’s shared conversation feature to ag- gregate real-world exchanges between developers and the model,

2

Table 1: Overview of DevGPT Dataset Statistics (Snapshot 20230831)

Statistic Count

Total number of unique conversations 2,345 Total number of conversations containing code 1,531 Total number of prompts and responses 17,9091 Total number of code snippets 11,751 Total number of Python code snippets 1,735 Total number of JavaScript code snippets 1,530

enriched with contextual metadata from platforms such as GitHub and Hacker News. DevGPT was constructed by systematically iden- tifying publicly shared ChatGPT links across these platforms. The dataset captures six snapshots collected between July 27, 2023, and August 31, 2023, ensuring temporal coverage and resilience against deleted or modified links. Each snapshot consolidates data from multiple software development artifacts, including issues, pull re- quests, commits, discussions, and external posts.

In total, the dataset comprises 17,909 prompts and responses, alongside 11,751 code snippets spanning multiple programming languages. The most frequent languages include Python, JavaScript, and Bash, reflecting common practices in developer tooling and scripting. Table 2 summarizes the top 10 code types by snippet count and their distribution across conversations.

This study focuses on the snapshot labeled 20230831, curated on October 23, 2023. This snapshot includes 2,891 shared Chat- GPT links, corresponding to 2,237 references across GitHub and Hacker News. After deduplication, 2,345 unique ChatGPT conver- sations were retained for analysis. Table 1 provides an overview of the selected snapshot. The dataset is organized into JSON files that encapsulate metadata such as conversation URLs, timestamps, and extracted HTML content. A consolidated CSV file indexes all shared ChatGPT links to facilitate querying and analysis, with full documentation available in the official repository [17].

For this research, the analysis was limited to Python and JavaScript snippets. This decision was based on two factors: first, both lan- guages are among the most widely adopted in the software industry, as reported in the Stack Overflow Developer Survey 2023 [18]; and second, they represent the largest share of generated code in the selected dataset snapshot (see Table 2).

Focusing on high-frequency languageswithin this curated dataset allows the study to identify patterns in AI-assisted coding practices, evaluate the quality of generated code, and observe how developers engage with AI tools in real-world contexts.

3.2 RQ1 RQ1: What quality issues are most commonly identified by linters in ChatGPT-generated code snippets?

This research question investigates the most prevalent quality issues present in ChatGPT-generated code. To address it, we ana- lyzed Python and JavaScript snippets extracted from the DevGPT dataset snapshot 20230831, focusing on these two languages due to their high prevalence within the dataset and their significance in the software development community [18]. Table 3 summarizes the total number of extracted Python and JavaScript snippets and their distribution across conversations.

Table 2: Top 10 Code Types by Snippet Count and the Number of Conversations they appear

Code Type Total Snippets Conversations

Python 1,735 317 JavaScript 1,530 256 Bash 1,435 265 Shell (sh) 659 370 HTML 615 111 CSS 554 119 C# 478 41 Java 345 64 SQL 246 66 TypeScript 240 63

Table 3: Extracted Code Snippets Prior to Preprocessing

Language # Code Snippets Conversations

Python 1,7342 317 JavaScript 1,530 199

Total 3,264 516

A custom Python script was used to parse the dataset’s JSON files and retrieve ChatGPT-generated code snippets together with associated metadata. The extraction pipeline included filtering for Python and JavaScript snippets, organizing them by conversation identifier, and applying basic filename sanitization to handle in- valid characters. To ensure that only meaningful and analyzable snippets were included, a cleaning step removed incomplete, trivial, or syntactically invalid code that could distort linter results.

A snippet was considered valid for analysis if it satisfied two criteria:

• Syntactic validity: The snippet must pass language-specific syntax checks.

• Structural completeness: The snippet must contain exe- cutable logic such as function or class definitions, beyond trivial imports.

Python snippets were validated using the compile function and analyzed via the ast module to classify them into Successful, Exe- cutable, Only Import, or Failed. JavaScript snippets underwent static analysis using Esprima, checking for syntax correctness, presence of function or class declarations, and balanced braces. Tables 4 and 5 summarize the outcome of the cleaning step, showing how many snippets were ultimately retained for analysis.

Table 4: JavaScript Code Snippet Cleaning Results

Description Count Total files with syntax errors 414 Total files with function or class definitions 475 Total files with balanced braces 1,112 Total files classified as complete 471

3

Table 5: Python Code Snippet Cleaning Results

Description Count Snippets with only executable statements 761 Snippets with only import statements 41 Snippets without function/class definition 4 Failed snippets (syntax errors) 164 Successful snippets 764

Table 6: Linters Utilized and Their Analytical Focus Areas

Linter Analytical Focus

Py th on

Pylint [8] Code standard enforcement and syntac- tic consistency

Flake8 [5] Style guide adherence and lightweight syntax checking

Bandit [1] Detection of security vulnerabilities

Ja va Sc rip

t

ESLint (+plugins) [4] Pattern checking, security analysis, and code quality enforcement

JSHint [7] Static error detection and potential bug identification

StandardJS [6] Strict style guideline enforcement

The cleaned Python and JavaScript snippets were then subjected to static analysis using multiple linters to systematically identify style violations, potential bugs, maintainability issues, and security vulnerabilities. For Python, we used Pylint [8], Flake8 [5], and Bandit [1]. Pylint and Flake8 focused on coding standards and stylistic consistency, while Bandit targeted security issues aligned with OWASP guidelines [19].

For JavaScript, we employed ESLint [4] with security-focused plugins such as eslint-plugin-security [3] and eslint-plugin-no-unsanitized [2], along with JSHint [7] and StandardJS [6]. ESLintwas configuredwith the eslint:recommended ruleset, while JSHint and StandardJS were applied using default con- figurations to ensure compatibility with modern JavaScript syntax. All linters were executed via automated scripts for batch processing, and their outputs were normalized into a common CSV format for further analysis.

Table 6 summarizes the linters and their primary focus areas. To enable a unified analysis across different tools and languages,

all reported warnings and errors were mapped into four categories:

(1) Code Style: Formatting, naming, and stylistic guideline vio- lations.

(2) Potential Bug: Patterns likely to cause runtime errors or unintended behavior.

(3) Code Smell: Indicators of poor design choices affecting maintainability.

(4) Code Vulnerability: Security-related issues posing poten- tial exploitation risks.

While some linters natively classify findings into these cate- gories, others (e.g., JSHint) required manual interpretation based on predefined heuristics to ensure consistent categorization across the dataset.

This preparation established a consistent dataset and linter con- figuration for answering RQ1. The actual findings of this analysis are presented in Section 4.1.

Code Style (e.g., C0114 – Missing Module Docstring) This warning from Pylint indicates that a module lacks a descriptive docstring at the top, which is important for maintainability and documentation.

# Code causing C0114 def add(a, b):

return a + b

Adding a module-level docstring resolves the issue:

""" Module for simple arithmetic operations.""" def add(a, b):

return a + b

Code Style (e.g., E501 – Line Length Limit) This Flake8 warn- ing flags lines that exceed the maximum recommended length, impacting readability and code formatting.

def long_line_example (): return "This␣is␣a␣very␣ long␣line␣of␣code␣that␣is␣likely␣to␣cause␣an␣ issue␣with␣the␣line␣length␣limit␣enforced␣by␣ PEP␣8␣and␣detected␣by␣Flake8 ..."

Breaking the line into multiple parts resolves the issue:

def long_line_example (): return "This␣is␣a␣very␣long␣line␣of␣code␣

broken␣into␣multiple␣lines␣" \ "to␣adhere␣to␣PEP␣8␣style␣guide␣

recommendations."

Potential Bug (e.g., E0602 – Undefined Variable) This Pylint error occurs when a variable is used before being defined, which can lead to runtime exceptions.

# Code causing E0602 def print_name ():

print(name) # 'name' is not defined before this usage

Fixing the bug requires defining the variable beforehand:

def print_name (): name = "Alice" print(name)

Code Vulnerability (e.g., B601 – Potential SQL Injection) Bandit flags dynamic SQL query construction using string concate- nation with user input, which can lead to SQL injection attacks.

# Vulnerable code causing B601 query = "SELECT␣*␣FROM␣users␣WHERE␣user_id␣=␣'" +

user_id + "'"

A safer, parameterized version avoids this risk:

query = "SELECT␣*␣FROM␣users␣WHERE␣user_id␣=␣%s", (user_id ,)

4

Analyzing the raw ChatGPT-generated snippets establishes a baseline understanding of the model’s intrinsic code quality before any developer intervention. This step isolates issues that originate solely from AI generation, ranging from simple style violations to potential maintainability and security concerns. We hypothesized that the majority of problems would be stylistic or convention- related rather than functional or security-critical, reflecting Chat- GPT’s tendency to produce syntactically correct but sometimes un- polished code. By categorizing and quantifying these issues across Python and JavaScript, RQ1 provides a reference point for evaluat- ing whether developer selection and reuse (addressed in RQ2) lead to higher quality or if problematic patterns persist into real-world projects.

This methodology provides the foundation for understanding the intrinsic quality of AI-generated code prior to reuse, enabling direct comparison with the quality of developer-selected snippets in RQ2 and their subsequent evolution analyzed in RQ3. The results of this analysis are presented in Section 4.1.

3.3 RQ2 RQ2: How does reused ChatGPT-generated code on GitHub per- form in terms of quality, as measured by linter-detected issues?

RQ2 shifts the focus from the intrinsic quality of raw ChatGPT- generated snippets (RQ1) to the quality of code that developers actually reuse in GitHub repositories. Unlike RQ1, which analyzed raw outputs in isolation, RQ2 examines code that has already un- dergone an implicit selection process by developers who chose to integrate it into real projects. This analysis helps determine whether reused AI-generated code is of inherently higher quality, or if de- velopers commit it without addressing critical quality concerns.

RQ2 builds directly on the filtered dataset of reused snippets from RQ1. From the original mappings, only Python (.py) and JavaScript (.js) files were considered, excluding non-executable formats such as Markdown or JSON files. To ensure meaningful analysis, only files that remained accessible on GitHub were included. After this filtering, 157 reused snippets (140 Python and 17 JavaScript) were retrieved. By focusing exclusively on already reused snippets, this stage isolates the code that developers deemed valuable enough to integrate into collaborative environments.

Quality analysis setup. The same linting configuration from RQ1 was reused to maintain comparability. Python files were analyzed using Pylint [8] and Bandit [1], while JavaScript files were an- alyzed with ESLint [4] (with security plugins) and JSHint [7]. To simplify interpretation, linter outputs were grouped into three categories:

• Errors: Direct functionality or execution issues. • Warnings: Potentially problematic patterns or poor prac- tices.

• Conventions: Stylistic deviations from coding standards. This grouping allows direct comparison between languages while capturing the balance between functional correctness and main- tainability concerns.

By analyzing the code quality of reused snippets, RQ2 addresses whether developers integrate AI-generated code “as-is” or if the implicit selection process improves its quality. We hypothesized that reused snippets might show fewer trivial issues compared to

raw outputs from RQ1, but could still exhibit deeper maintainability and style violations—particularly for Python, which dominates the dataset. For JavaScript, fewer detected issues were expected due to both the smaller sample size and the more permissive default ESLint configuration.

This pipeline enabled a direct quality assessment of reused ChatGPT- generated snippets. The results of this analysis are presented in Section 4.2.

3.4 RQ3 RQ3: How do developers modify reused ChatGPT-generated code on GitHub over time?

RQ3 builds on RQ2 by shifting from static quality assessment to the evolution of reused snippets after integration into real projects. While RQ2 examined the quality of reused ChatGPT-generated code at the moment it was committed, RQ3 investigates how this quality changes over time. Specifically, it analyzes whether developers im- prove the code by resolving existing issues, inadvertently introduce new ones while adapting it, or leave quality problems unchanged. Understanding these modification patterns reveals whether AI- generated code becomes more maintainable after reuse or continues to propagate quality debt in collaborative repositories.

To observe how reused code evolves, we analyzed the commit history of snippets selected in RQ2. Only files with at least two commits were considered, as single-commit snippets could not reveal any change over time. Using the GitHub API, we retrieved each file’s full commit history, extracted the earliest (initial) and latest (final) versions, and prepared them for comparison. This process yielded 94 snippets suitable for longitudinal analysis: 86 in Python and 8 in JavaScript. Python dominates the sample, reflecting its higher prevalence in both ChatGPT-generated outputs and real- world reuse.

Both the initial and final versions of each snippet were analyzed using the same static analysis tools applied in RQ2. Python files were processed with Pylint [8] and Bandit [1], while JavaScript files were checked with ESLint [4] and JSHint [7]. Linter-reported issues were classified into three categories:

• Fixed: Issues present initially but absent in the final version. • Introduced: Issues absent initially but appearing in the final version.

• Unchanged: Issues persisting across both versions.

This classification allowed us to quantify net quality changes for each snippet, measuring whether developer modifications ulti- mately improved or degraded code quality.

We hypothesized that developers would primarily fix stylistic and trivial issues while leaving more complex maintainability or security concerns unresolved. At the same time, we expected that functional adaptations—such as integrating a snippet into a larger codebase—might inadvertently introduce new problems, potentially resulting in a mixed or even negative net quality outcome. Given Python’s stricter linting rules and larger sample size, we antici- pated more extensive modification patterns for Python compared to JavaScript, where smaller snippets and permissive ESLint defaults might result in minimal detected changes.

5

This methodology enabled a focused analysis of how developers refine or alter AI-generated code during integration and mainte- nance in collaborative environments. The results of this analysis are presented in Section 4.3.

4 STUDY EXECUTION This section discusses the execution of the project, the results ob- tained, and how they address the claims made in the introduction. The results are organized by research question (RQ1, RQ2, and RQ3) to evaluate the quality and modification patterns of ChatGPT- generated code snippets. The findings justify the design choices and assess the contributions of different aspects toward the overall goals of understanding AI-generated code in collaborative software development.

4.1 RQ1: Common Quality Issues in ChatGPT-Generated Code Snippets

RQ1: What quality issues are most commonly identified by linters in ChatGPT-generated code snippets? To address RQ1, we analyzed sanitized Python and JavaScript snip- pets from the DevGPT dataset (Snapshot 20230831). The analysis employed Pylint, Flake8, and Bandit for Python, and ESLint, JSHint, and StandardJS for JavaScript. After data cleansing, 764 Python snippets and 471 JavaScript snippets remained eligible for analysis. All reported issues were categorized into four groups: Code Style, Code Smell, Potential Bug, and Code Vulnerability, as described in Section 3.2.

The prevalence of each issue category was quantified as the proportion of affected snippets relative to the total number of ana- lyzed files. Tables 7 and 8 summarize these results for Python and JavaScript respectively.

Table 7: Python Snippet Quality Issues (764 Snippets)

Category Total Issues Files Affected Percentage Affected

Code Style 4,951 659 86.3% Potential Bug 4,147 353 46.2% Code Smell 355 ≥76 ≥10.0% Code Vulnerability 182 39 5.1%

Table 8: JavaScript Snippet Quality Issues (471 Snippets)

Category Total Issues Files Affected Percentage Affected

Code Style 26,722 468 99.4% Potential Bug 3,874 324 68.8% Code Smell 270 128 27.2% Code Vulnerability 14 12 2.5%

For Python snippets, Code Style violations were the most preva- lent, affecting 86.3% of files (659 out of 764). The most frequent issues were overly long lines (E501, 1,307 occurrences) and miss- ing module or function docstrings (C0114, 411 occurrences), both flagged by Flake8 and Pylint. Potential Bugs were detected in 46.2% of snippets, with undefined variables (E0602, 1,278 occurrences)

being the primary source. Code Smells were less common, esti- mated to affect roughly 10% of snippets, primarily involving poor design indicators such as excessive function arguments. Code Vul- nerabilities were rare, appearing in only 5.1% of snippets; Bandit flagged occasional risks such as missing request timeouts (B113, 57 occurrences).

For JavaScript snippets, Code Style violations were nearly uni- versal, affecting 99.4% of files (468 out of 471). StandardJS’s strict formatting rules generated the majority of these findings, includ- ing indentation inconsistencies (10,469 occurrences) and missing semicolons (9,592 occurrences). Potential Bugs appeared in 68.8% of snippets, with no-undef (1,353 instances) and no-unused-vars (413 instances) being the most common. Code Smells affected 27.2% of snippets, often triggered by ESLint’s no-console rule (263 occur- rences). Code Vulnerabilities were minimal (2.5%), with only rare in- stances such as unsafe prototype usage (no-prototype-builtins, 2 cases).

Across both languages, Code Style issues dominated, indicating that ChatGPT-generated code, while syntactically correct, lacks adherence to common project-level formatting and documenta- tion standards. Potential Bugs were also significant, particularly undefined or unused variables, highlighting the need for additional validation before reuse. Code Smells and Vulnerabilitieswere less fre- quent, likely because most generated snippets are relatively small and lack complex architectural structures or security-sensitive logic.

RQ1. What quality issues are most commonly identified by linters in ChatGPT-generated code snippets? Answer: The most common quality issues in ChatGPT- generated code snippets are Code Style violations, followed by Potential Bugs. Specifically, Python code exhibited frequent problems such as overly long lines (E501) and missing docstrings (C0114), while JavaScript snippets almost universally failed style checks under StandardJS, highlighting indentation and semicolon inconsistencies. Potential Bugs, primarily undefined and unused variables, were also widely detected in both languages. In con- trast, Code Smells and Security Vulnerabilities appeared far less frequently, likely due to the simplicity of most generated snip- pets. Implications: These findings imply that ChatGPT prioritizes functional correctness but neglects maintainability and stylistic conventions, resulting in code that cannot be directly integrated into production environments without further refinement. De- velopers must therefore invest additional effort in refactoring and linting before reuse. This baseline assessment establishes the intrinsic quality profile of AI-generated code and serves as a reference for RQ2, which evaluates whether developer selection and reuse lead to higher quality or propagate these deficiencies.

4.2 RQ2: Quality of Reused ChatGPT-Generated Code

RQ2: How does reused ChatGPT-generated code on GitHub per- form in terms of quality, as measured by linter-detected issues? To address RQ2, we analyzed 157 reused snippets (140 Python and 17 JavaScript) retrieved from GitHub repositories. These snippets, selected through developer reuse, were expected to exhibit higher

6

Figure 1: Top Issues from RQ2

quality than raw ChatGPT outputs. The same linting pipeline from RQ1 was applied to assess whether developers addressed style, maintainability, and bug-related issues prior to integration.

In total, 2,849 linter-detected issues were reported, all originating from Python snippets. The 17 JavaScript snippets yielded no ESLint or JSHint findings under default configurations. Table 9 summarizes the overall statistics, and Table 10 shows the issue breakdown by severity and most frequent rules.

Table 9: Overall Linter Issue Statistics

Metric Value

Total Snippets Analyzed 157 Python Snippets 140 JavaScript Snippets 17 Total Linter Issues 2849 Average Issues per Snippet 18.1 Median Issues per Snippet 18.0 Maximum Issues per Snippet 63 Minimum Issues per Snippet 0

The results implymoderate dispersion in quality, with an average of 18.1 issues per snippet, a median of 18.0, and a maximum of 63. Python snippets averaged 20.4 issues, whereas JavaScript snippets averaged none—they are either flawless or free from noted issues.

The 140 Python snippets displayed assorted issues: • Conventions (1,690 instances): – C0301: line-too-long (764): Lines longer than 79 characters, such as a single line of a complicated list comprehension, which negatively affects readability.

– C0116: missing-function-docstring (411): Functions with- out a descriptive docstring, e.g., a utility function with no explanation regarding what parameters it takes.

– C0103: invalid-name (128): Non-standard names, that is, one-character variables such as x instead of something like counter.

• Errors (203 instances): – E0401: import-error (191): Missing imports, i.e., import pandas without checking that the library is installed.

– E0606: potentially-used-before-assignment (10): Using vari- ables before some assignment, for example, assignment conditionally.

• Warnings (723 instances): – W0621: redefined-outer-name (287): Shadowing of outer variables, i.e., an outer loop variable reused in an inner scope.

– W0611: unused-import (100): Unnecessary imports, such as import math and not utilizing it.

These findings indicate that even reused Python code retains many stylistic andmaintainability issues, with occasional functional errors that remain unaddressed.

The 17 JavaScript snippets produced no ESLint, JSHint, or Stan- dardJS issues under default configurations. This outcome is likely influenced by three factors: the very small sample size, the minimal nature of many JavaScript snippets (e.g., single-line console.log statements), and the permissive nature of default ESLint and JSHint configurations. As a result, the JavaScript results cannot be consid- ered conclusive, and Python findings dominate the interpretation of reused snippet quality.

Compared to raw ChatGPT outputs from RQ1, reused Python snippets in RQ2 did not exhibit markedly improved quality. Style violations and trivial errors remained widespread, suggesting devel- opers often integrate ChatGPT code with minimal refactoring. The absence of meaningful JavaScript findings further limits evidence of active quality filtering during reuse. In other words, developers appear to select snippets based on immediate functional utility rather than their maintainability or adherence to coding standards.

RQ2.How does reused ChatGPT-generated code on GitHub perform in terms of quality, as measured by linter- detected issues? Answer: In answer to RQ2, most reused ChatGPT-generated snippets in public repositories are not pre-cleaned for linter- detected issues; developers typically reuse them without ad- dressing style, maintainability, or even some functional concerns. Python snippets averaged 18.1 issues per file, dominated by con- vention violations and minor warnings, while JavaScript reuse was too limited to provide meaningful quality insight. Implications: These findings imply that the act of developer selection does not substantially improve AI-generated code qual- ity before reuse. Instead, ChatGPT outputs—and their associated issues—are propagated into shared repositories with minimal modification. This highlights a disconnect between the rapid prototyping benefits of AI code generation and the quality ex- pectations of collaborative environments. For practitioners, this underscores the need for mandatory linting and quality gates before integrating AI-generated code. For AI tool developers, it emphasizes the importance of producing lint-compliant code to reduce the post-generation burden on developers.

7

Table 10: Linter Issues by Severity and Top Examples

Severity Count Percentage Top Issues (Code: Description, Count)

Errors 203 7.1% E0401: import-error, 191; E0606: possibly-used-before-assignment, 10; E0602: undefined-variable, 2

Conventions 1690 59.3% C0301: line-too-long, 764; C0116: missing-function-docstring, 411; C0103: invalid-name, 128

Warnings 723 25.4% W0621: redefined-outer-name, 287; W0611: unused-import, 100; W1203: logging-fstring-interpolation, 52

Total 2849 100%

4.3 RQ3: Developer Responses to Linter-Detected Issues in ChatGPT-Generated Code

RQ3. How do developers modify reused ChatGPT-generated code on GitHub over time? RQ3 investigates whether developers improve the quality of reused ChatGPT-generated snippets over time when integrating them into real projects. Building on RQ2, which showed that developers often reuse snippets without fixing existing linter- detected issues, RQ3 examines whether such issues are later re- solved or persist—and whether additional issues are introduced—by analyzing snippet evolution across commit history. Specifically, we compared the earliest (initial) and most recent (final) versions of 94 multi-commit snippets (86 Python and 8 JavaScript) from the DevGPT dataset.

For each snippet, we quantified three types of changes in linter- detected issues: fixed (present initially but removed in the final version), introduced (newly appearing in the final version), and unchanged (persisting across both versions). Python files were ana- lyzed with Pylint, Flake8, and Bandit, while JavaScript files were checked with ESLint, JSHint, and StandardJS.

Across the dataset, developers fixed 1,045 issues, introduced 1,612 new ones, and left 144 unchanged, resulting in an average of 11.12 fixes, 17.15 introductions, and 1.53 unchanged issues per snippet. This leads to a net quality decline of –6.03 issues per snippet. While developers clearly engage with reused code, their modifications often create more problems than they resolve. To ensure accuracy, linter outputs were cross-validated using multiple tools per language, and a subset of high-change snippets was man- ually reviewed to confirm legitimacy. Table 11 summarizes these findings.

Table 11: Summary of Linter-Detected Issue Changes

Metric Value

Total Snippets 94 Total Fixed Issues 1,045 Total Introduced Issues 1,612 Total Unchanged Issues 144 Average Fixed per Snippet 11.12 Average Introduced per Snippet 17.15 Average Unchanged per Snippet 1.53 Net Change per Snippet -6.03

Python snippets showed clear modification activity. Developers fixed 1,045 issues (12.15 per snippet), introduced 1,612 issues (18.74 per snippet), and left 144 unchanged (1.67 per snippet). This pattern suggests active refactoring and integration work, but also high- lights how functional adaptations—such as embedding a snippet into a larger project—frequently introduce new style violations or maintainability issues.

In contrast, the 8 JavaScript snippets exhibited no measurable changes: zero fixed, introduced, or unchanged issues. This static result reflects the very small sample size, the trivial nature of most JavaScript snippets (often single-line console statements), and per- missive default ESLint/JSHint configurations. Consequently, no meaningful conclusions about JavaScript modification behavior can be drawn; Python findings dominate the analysis.

Table 12: Language-Specific Changes in Linter Issues

Metric Python JavaScript

Snippets 86 8 Total Fixed 1,045 0 Total Introduced 1,612 0 Total Unchanged 144 0 Average Fixed per Snippet 12.15 0.0 Average Introduced per Snippet 18.74 0.0 Average Unchanged per Snippet 1.67 0.0

Certain snippets highlighted diverse developer responses to lin- ter issues:

Table 13: Notable Modification Patterns in Reused Snippets

Metric Bounding Box Alignment3 Vectorizer Error4

Fixed issues 30 29 Introduced issues 28 61 Unchanged issues 13 2 Net change +2 –32

Table 13 highlights two representative Python snippets. The first was a utility function for aligning bounding boxes in an image- processing pipeline. Developers fixed 30 issues, mainly missing docstrings and unused imports, but also introduced 28 new style violations, resulting in a modest net improvement of two issues.

8

Figure 2: Changes in Linter-Detected Issues for Python5

The second snippet adapted a text vectorizer for feature extraction. Although 29 issues were resolved, 61 new ones were introduced, mostly variable shadowing, leading to a net decline of 32 issues. These examples show that even targeted refinements can lead to additional technical debt when functional changes take priority over maintainability.

Figure 2 visualizes the distribution of fixed, introduced, and unchanged issues, highlighting Python’s modification activity.

The net decline in quality can be attributed to several factors. Developers typically prioritize making the snippet work within a larger project rather than maintaining linting standards. The lack of context or documentation in ChatGPT-generated code makes it harder to modify safely, leading to unintended errors. Time con- straints also encourage fixing only critical runtime issues while leaving non-blocking style or maintainability violations unresolved. In the case of JavaScript, the absence of observed changes does not indicate higher quality but instead reflects the limited dataset and trivial snippet complexity.

RQ3. How do developers modify reused ChatGPT-generated code on GitHub over time? Answer: Developers do modify reused ChatGPT-generated snippets over time, but these changes do not consistently improve their quality. While 1,045 pre-existing issues were fixed, 1,612 new issues were introduced, resulting in a net de- cline of 6.03 issues per snippet. Python snippets showed active but counterproductive modification patterns, while JavaScript snippets provided no meaningful insight due to their limited size and triviality. Implications: These findings indicate that post-reuse modifica- tions often prioritize functional integration over maintainability, leading to an accumulation of new technical debt rather than overall quality improvement. For developers, this highlights the need for deliberate review and quality assurance when adapting AI-generated code. For AI tool developers, it underscores the importance of generating context-aware, lint-compliant code to reduce the downstream burden of refactoring. Combined with RQ1 and RQ2, these results reveal that quality problems in AI- generated code not only persist but can worsen over time in collaborative repositories.

5 DISCUSSION This section deals with research results regarding the reuse, quality, and modification of ChatGPT-generated code on GitHub. These results answer the three major research questions. Every subsec- tion presents a synthesis of what was found and its interpretation and relates these to the literature, allowing for a deeper analysis focusing on practical and theoretical consequences.

5.1 RQ1: Reuse of ChatGPT-generated code on GitHub

RQ1 asked: What quality issues are most commonly identified by linters in ChatGPT-generated code snippets? The analysis revealed that style-related violations dominated across both Python and JavaScript, affecting 86.3% of Python snippets and 99.4% of JavaScript snippets. Typical problems included overly long lines, missing doc- strings, and non-standard variable names. Potential bugs were the second most frequent category, appearing in nearly half of Python snippets and two-thirds of JavaScript snippets, with undefined or unused variables as the most common examples. In contrast, code smells were relatively uncommon, and security vulnerabilities were rare, affecting fewer than 5% of all snippets.

These findings suggest that ChatGPT prioritizes functional plau- sibility over strict adherence to style guides and maintainability conventions. This aligns with prior work by Sobania et al. [22], who observed that LLM-generated code tends to be syntactically correct but often fails to meet community-specific coding standards. Simi- larly, Liu et al. [16] noted that large language models produce code that may execute successfully but lacks the consistency and read- ability needed for long-term maintainability. The relative absence of code smells and vulnerabilities in this study can be explained by the simplicity of the analyzed snippets: most were short, self- contained examples that did not involve complex architectural logic or security-sensitive functionality.

9

From a practical perspective, these results highlight that ChatGPT- generated code can be useful as a starting point but requires ad- ditional processing before integration into collaborative environ- ments. Developers must apply automated linters or formatters to ensure compliance with project standards, and human review re- mains essential for maintainability. For AI tool designers, these findings underscore the need for integrated style and linting checks during code generation. If models could generate lint-compliant code by default, this would reduce the manual cleanup required by developers.

This study has some limitations. The analysis relied on a single dataset (DevGPT snapshot 20230831) and focused only on Python and JavaScript, which may not represent other programming lan- guages or broader development contexts. Additionally, the severity of issues was evaluated only through static linters; runtime or context-specific problems were outside the scope of this research. Future work could expand this analysis to larger, multi-language datasets and investigate how different prompt styles or model ver- sions influence the types of issues produced.

Overall, RQ1 establishes a baseline quality profile of ChatGPT- generated code, showing that while it is generally functional, it is far from production-ready in terms of maintainability and style. These findings provide important context for RQ2, which examines whether developers improve these quality issues before integrating the snippets into public repositories.

5.2 RQ2: Quality of Reused ChatGPT-Generated Code

RQ2 asked: How does reused ChatGPT-generated code on GitHub per- form in terms of quality, as measured by linter-detected issues? The analysis revealed that reused snippets still contained a substantial number of quality issues, with 2,849 problems detected across 157 files—an average of 18.1 issues per snippet. Most of these were style and convention violations (59.3%), such as overly long lines and missing docstrings. Warnings accounted for 25.4% of issues, while functional errors like unresolved imports made up 7.1%. This indi- cates that although the reused code is generally executable, it does not align with established coding standards and maintainability expectations.

These findings align with prior research by Liu et al. [15], who observed that while large language models like ChatGPT often produce syntactically correct code, they frequently fail to adhere to best practices, leaving developers to address stylistic inconsisten- cies and dependency issues. Similarly, Siddiq et al. [21] found that AI-generated code often lacks contextual awareness, resulting in un- resolved imports and other integration problems—issues also seen in this predominantly Python-based dataset. In contrast, JavaScript snippets showed no reported issues, but this result is likely mislead- ing rather than indicative of higher quality. The small sample size (17 files) and permissive ESLint default configuration likely explain the absence of warnings. This aligns with observations by Fajkovic and Rundberg [12], who highlighted how language-specific linter configurations and generation tendencies can produce inconsistent results across programming languages.

Taken together, these results indicate that developer selection alone does not significantly improve the quality of AI-generated

code before reuse. The reused snippets on GitHub were not pre- cleaned, suggesting that developers adopt ChatGPT outputs directly for functional purposes without investing in style or maintainability refinements. As a result, the same style, maintainability, and minor functional issues identified in raw ChatGPT-generated code persist into public repositories. This finding has important implications for development workflows. While ChatGPT can accelerate proto- typing, the subsequent quality assurance effort required to bring the code up to production standards may offset initial productivity gains, as also suggested by Chen et al. [10].

This study has some limitations. The results are based on a single snapshot of reused snippets and focused only on Python, as Javascript didn’t have any meaningfull results.. Other languages may show different issue profiles, and different linter configurations could yield alternative findings. Additionally, the analysis only covers static quality metrics; runtime or project-specific integration issues were outside the scope. Future research could expand the dataset across more repositories and explore how prompt design or fine-tuned models affect the quality of reused snippets.

In conclusion, RQ2 demonstrates that reused ChatGPT-generated code on GitHub remains of moderate quality and requires addi- tional polishing. Developers appear to prioritize functional reuse over maintainability, which raises the need for mandatory quality gates before integration. For AI tool developers, these results high- light the importance of generating lint-compliant, context-aware code to reduce the downstream burden of manual cleanup. These findings also set the stage for RQ3, which examines whether devel- opers improve these quality issues over time or if they persist and accumulate.

5.3 RQ3: Modifications to Reused ChatGPT-Generated Code

RQ3 asked: How do developers modify reused ChatGPT-generated code on GitHub over time? The longitudinal analysis of 94 multi- commit snippets revealed that developers actively modify reused code but do not consistently improve its quality. Across all snippets, 1,045 issues were fixed, 1,612 new issues were introduced, and 144 remained unchanged, resulting in a net decline of 6.03 issues per snippet. This means that while developers attempt to refine AI- generated code, their changes often introduce additional problems, leading to a gradual erosion of maintainability as measured by linter-detected issues.

These findings mirror observations by Das et al. [11], who re- ported that developer customization of AI-generated code frequently introduces new errors, particularly when adapting snippets to spe- cific project contexts. Similarly, Sobania et al. [22] noted that in- tegrating AI-generated code into larger systems tends to increase complexity, which can create new style or structural violations even when resolving original defects. This dynamic was evident in the contrasting case examples. One snippet involving bounding box alignment showed modest improvement, with 30 issues fixed and 28 new ones introduced, yielding a slight net gain of two issues. In contrast, a snippet addressing a text vectorizer error saw 29 issues resolved but 61 new ones added, producing a sharp decline of 32 issues. Such variability underscores that outcomes depend heavily on the context of reuse and the nature of modifications.

10

The persistence of 144 unchanged issues also highlights a de- gree of developer pragmatism. As Watanabe et al. [23] observed in their study of AI-assisted code reviews, developers often pri- oritize functional integration over minor style or convention vi- olations, particularly when under time or resource constraints. Language-specific findings also point to differences in how reused code evolves. Python snippets exhibited active modification pat- terns, averaging 12.15 fixes and 18.74 introductions per snippet, whereas JavaScript snippets showed no measurable changes. This discrepancy likely stems from the very small JavaScript sample size and the triviality of those snippets, which often consisted of simple, single-line statements, as well as the permissiveness of ESLint’s default configuration. Hansson and Ellréus [14] similarly argue that language-specific tendencies in both code generation and de- veloper workflows necessitate tailored AI tools to minimize such inconsistencies.

From a broader perspective, these findings emphasize that AI- assisted code reuse carries a dual nature. While developers modify AI-generated code to fit functional needs, these changes often in- troduce new technical debt rather than reducing it. This suggests a need for better integration support, such as AI-assisted refactoring tools or lint-aware code generation pipelines, to prevent quality degradation during post-reuse adaptation. For development prac- tices, it highlights the importance of treating AI-generated code as an initial draft rather than a production-ready solution, requiring deliberate review and structured quality assurance before long-term integration.

There are some limitations. The dataset analyzed was limited to a single snapshot (DevGPT 20230831), and the sample of multi- commit JavaScript snippets was too small to draw meaningful con- clusions for that language. Additionally, linter-based analysis cap- tures only static, surface-level quality metrics and does not assess runtime errors or deeper architectural maintainability issues. Future research could examine larger datasets across more languages and investigate how different project environments or developer work- flows influence modification patterns. Qualitative studies, such as developer interviews or surveys, could further clarify why certain issues are fixed while others are tolerated.

In conclusion, RQ3 demonstrates that while developers do in- vest effort in modifying reused ChatGPT-generated code, these changes do not consistently improve quality and may even worsen it. Functional integration often takes precedence over maintainabil- ity, reinforcing the need for AI tools that generate cleaner, more context-aware code and for workflows that integrate automated quality gates during adaptation.

6 LIMITATIONS This study has several limitations that should be considered when interpreting the results. A first limitation concerns the scope of the dataset. The analysis was based on a single snapshot of reused snippets (DevGPT 20230831), which may not fully capture all pat- terns of AI-generated code reuse on GitHub. In particular, the small number of multi-commit JavaScript snippets restricted meaningful interpretation for that language and may have biased the findings toward Python-specific behaviors. A second limitation lies in the exclusive use of static linter-based

analysis. While linters are effective at detecting style violations, minor bugs, and some security risks, they cannot capture runtime errors, deeper architectural maintainability concerns, or the func- tional intent behind developer modifications. As a result, the quality changes observed may not reflect the complete impact of developer edits in real project contexts. A third limitation relates to the lack of qualitative context. This study inferred developer priorities—such as emphasizing functional integration over stylistic consistency—only from observed modi- fication patterns. Without direct input from developers through interviews or surveys, these interpretations remain assumptions rather than confirmed motivations. Despite these limitations, the findings provide valuable insight into how reused ChatGPT-generated code evolves in collaborative repositories. Future work could address these gaps by analyzing larger and more diverse datasets across multiple programming lan- guages, incorporating runtime and architectural quality metrics, and combining quantitative data with qualitative insights from de- velopers to better explain why certain issues are fixed while others are tolerated.

7 THREATS TO VALIDITY In addition to the limitations discussed above, this study is sub- ject to several potential threats to validity that may influence the interpretation of its findings.

Internal validity. The analysis relies on automated scripts for snippet extraction, cleaning, and linter execution. Anymisclassifica- tion or parsing errors during these preprocessing steps could affect the accuracy of the results. Additionally, the absence of original local snippets for some cases forced reliance on repository versions, which may have introduced discrepancies between the generated and reused code.

Construct validity. Static linters were used as the primary proxy for code quality. While they are widely adopted and effective for detecting style violations and certain types of maintainability issues, they cannot fully capture runtime errors, semantic correct- ness, or deeper architectural concerns. Thus, the study may not fully represent the broader concept of “code quality” as perceived in real-world development.

External validity. The dataset was limited to a single DevGPT snapshot (20230831) and focused exclusively on Python and JavaScript. This may limit the generalizability of the findings to other program- ming languages, repository ecosystems, or newer versions of Chat- GPT. Furthermore, the small number of multi-commit JavaScript snippets prevents drawing robust conclusions for that language.

Conclusion validity. While the study quantifies changes in linter-detected issues, it does not assess whether these changes significantly impact the functional reliability of the code. Similarly, interpretations about developer priorities (e.g., favoring functional integration over style) are inferred from patterns in the data rather than direct qualitative evidence.

Recognizing these threats helps contextualize the results and highlights areas for future improvement. Future work could miti- gate these risks by using additional quality metrics (e.g., dynamic analysis or human code review), expanding the dataset across more languages and repositories, and triangulating quantitative findings

11

with qualitative data from developers to better validate the observed modification patterns.

8 RELATEDWORK Recent studies have examined how large language models (LLMs) generate code and the challenges of integrating it into real-world projects. Chen et al. [10] systematically evaluated OpenAI’s Codex on algorithmic benchmarks, showing strong performance on small, isolated tasks but a marked decline as contextual complexity in- creased. Sobania et al. [22] reached similar conclusions when com- paring ChatGPT’s bug-fix suggestions to human patches, noting that AI outputs can introduce new vulnerabilities when applied to larger, collaborative systems. Unlike these controlled, in-lab eval- uations, our work follows ChatGPT-generated code end-to-end within GitHub repositories, capturing how developers reuse and adapt snippets over time.

A parallel line of research focuses on static quality analysis and code smells. Azeem et al. [9] demonstrated that linters like Pylint and ESLint are effective at flagging low-level style violations but struggle to detect deeper semantic anti-patterns. Liu et al. [16] further showed that LLM-generated code frequently contains de- pendency issues such as missing imports and unsafe API usage. We build on these findings by quantifying how often such errors persist into public repositories, revealing that unresolved imports affect over 40% of reused Python snippets. Siddiq et al. [21] ob- served a similar need for “post-generation cleanup” in LLM-written JUnit tests, which required manual tuning to align with project con- ventions. Our work extends this insight beyond testing, analyzing general-purpose Python and JavaScript snippets and tracking how their quality changes after developer intervention.

Finally, several studies investigate the adoption and human in- tegration of AI-generated code. Yetiştiren et al. [26] surveyed AI- assisted code usage across GitHub, highlighting developer skepti- cism and patchy quality as barriers to widespread uptake. Grewal et al. [13] examined pull requests containing ChatGPT-generated code and found that substantial manual editing is the norm before acceptance. Our study refines these observations by providing fine- grained, linter-based measurements: we report an 11.6% reuse rate for ChatGPT snippets, analyze the types of issues most commonly present at the time of reuse, and track how developer modifications affect code quality over multiple commits.

Taken together, prior research has identified that LLM-generated code can be syntactically correct but often lacks maintainability, requires manual adaptation, and faces limited adoption due to qual- ity concerns. However, little is known about how these quality issues persist, evolve, or worsen after integration into real-world repositories. Our work addresses this gap through a longitudinal analysis of reused ChatGPT-generated code on GitHub, combining reuse rates, static quality metrics, and modification patterns to pro- vide a comprehensive view of AI-assisted software development in practice.

9 CONCLUSION This thesis investigated the reuse, quality, and evolution of ChatGPT- generated code within public GitHub repositories. By addressing three research questions, it provided a layered understanding of

how AI-generated snippets integrate into real-world projects, how their quality compares at different stages of reuse, and how devel- opers modify them over time.

For RQ1, the analysis of raw ChatGPT-generated snippets re- vealed that while the code is generally syntactically correct, it exhibits substantial style violations and minor maintainability con- cerns. Python and JavaScript snippets alike were dominated by convention-related issues such as missing docstrings, overly long lines, and inconsistent formatting. Potential bugs, including unde- fined or unused variables, were also common, whereas deeper code smells and security vulnerabilities were comparatively rare due to the small, self-contained nature of most snippets. These results show that ChatGPT prioritizes functional plausibility over strict adherence to style guides or long-term maintainability, making its outputs useful for prototyping but not immediately ready for production environments.

For RQ2, the evaluation of reused snippets in GitHub reposito- ries demonstrated that developer selection alone does not substan- tially improve AI-generated code quality. Even after integration into collaborative projects, reused Python snippets averaged 18.1 linter-detected issues per file, with the majority being style and maintainability violations that remained unaddressed. JavaScript snippets yielded no meaningful results due to the small sample size and permissive linter settings. This indicates that developers often adopt ChatGPT outputs directly for their functional value without dedicating additional effort to refactoring, thereby propagating the same deficiencies found in raw outputs into public repositories.

RQ3 expanded the analysis to consider how reused snippets evolve across commits. While developers fixed 1,045 existing is- sues, they simultaneously introduced 1,612 new ones, resulting in a net decline of 6.03 issues per snippet. Python snippets showed active but counterproductive modification patterns, where func- tional adaptations—such as embedding snippets into larger code- bases—frequently introduced new style or structural problems. In contrast, the very limited JavaScript sample exhibited no measur- able changes. These findings suggest that post-reuse modifications tend to prioritize immediate functionality over maintainability, lead- ing to an accumulation of technical debt rather than sustained quality improvement.

Taken together, these findings highlight three key implications. First, ChatGPT-generated code serves as a practical starting point for developers but cannot yet be considered production-ready with- out further refinement. Second, the lack of quality filtering during reuse means that problematic patterns persist into collaborative repositories, potentially undermining maintainability. Third, even when developers actively modify AI-generated code, their changes often introduce new issues, underscoring the need for better inte- gration support. To mitigate these challenges, future AI-assisted development workflows should integrate lint-aware code gener- ation, provide clearer contextual metadata to reduce integration complexity, and adopt stricter automated quality gates during reuse and adaptation.

Future research should expand the analysis to more program- ming languages and larger, longitudinal datasets while complement- ing static linter metrics with runtime and architectural assessments. Qualitative studies, such as developer interviews or surveys, could further clarifywhy certain issues are fixedwhile others are tolerated.

12

Exploring prompt-engineering techniques and model fine-tuning for linter compliance could also improve the baseline quality of generated snippets, reducing the downstream burden on developers. By aligning AI generation more closely with established coding conventions and real-world project constraints, it may become pos- sible to bridge the gap between rapid AI-assisted prototyping and maintainable, production-ready software.

REFERENCES [1] [n.d.]. Bandit Documentation. https://bandit.readthedocs.io/en/latest/. Accessed:

2023-11-29. [2] [n.d.]. eslint-plugin-no-unsanitized. https://github.com/mozilla/eslint-plugin-

no-unsanitized. Accessed April 2024. [3] [n.d.]. eslint-plugin-security. https://github.com/nodesecurity/eslint-plugin-

security. Accessed April 2024. [4] [n.d.]. ESLint Rules Reference. https://eslint.org/docs/rules/. Accessed: 2023-11-

29. [5] [n.d.]. Flake8 Rules. https://www.flake8rules.com/. Accessed: 2023-11-29. [6] [n.d.]. JavaScript Standard Style. https://standardjs.com/rules.html. Accessed:

2023-11-29. [7] [n.d.]. JSHint Documentation. https://jshint.com/docs/. Accessed: 2023-11-29. [8] [n.d.]. Pylint Rules. https://pylint.pycqa.org/en/latest/technical_reference/

features.html. Accessed: 2023-11-29. [9] M. I. Azeem, F. Palomba, L. Shi, and Q. Wang. 2019. Machine learning tech-

niques for code smell detection: A systematic literature review and meta- analysis. Information and Software Technology 108 (2019), 115–138. https: //doi.org/10.1016/j.infsof.2018.12.009

[10] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fo- tios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shan- tanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021. Evaluating large language models trained on code. https://doi.org/10.48550/arXiv.2107.03374 arXiv:2107.03374 [cs.LG]

[11] J. K. Das, S. Mondal, and C. K. Roy. 2025. Why Do Developers Engage with Chat- GPT in Issue-Tracker? Investigating Usage and Reliance on ChatGPT-Generated Code. In 2025 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 68–79. https://doi.org/10.1109/SANER64311.2025. 00015

[12] E. Fajkovic and E. Rundberg. 2023. The impact of AI-generated code on web development: a comparative study of ChatGPT and GitHub Copilot. https://www. diva-portal.org/smash/record.jsf?pid=diva2%3A1769082.

[13] B. Grewal, W. Lu, S. Nadi, and C. P. Bezemer. 2024. Analyzing Developer Use of ChatGPT-Generated Code in Open Source GitHub Projects. In Proceedings of the 21st International Conference on Mining Software Repositories. 157–161. https://doi.org/10.1145/3643991.3645072

[14] E. Hansson and O. Ellréus. 2023. Code correctness and quality in the era of AI code generation: Examining ChatGPT and GitHub Copilot. https://www.diva- portal.org/smash/record.jsf?pid=diva2%3A1764568.

[15] J. Liu, C. S. Xia, Y. Wang, and L. Zhang. 2023. Does ChatGPT gen- erate your code correctly? Rigorous evaluation of large language models for code generation. In Advances in Neural Information Processing Systems, Vol. 36. 21558–21572. http://proceedings.neurips.cc/paper_files/paper/2023/ hash/43e9d647ccd3e4b7b5baab53f0368686-Abstract-Conference.html

[16] Y. Liu, T. Le-Cong, R. Widyasari, C. Tantithamthavorn, L. Li, X. B. D. Le, and D. Lo. 2024. Refining ChatGPT-generated code: Characterizing and mitigating code quality issues. ACM Transactions on Software Engineering and Methodology 33, 5 (2024), 1–26. https://doi.org/10.1145/3643674

[17] NAIST Software Engineering Lab. 2024. DevGPT Dataset. https://github.com/ NAIST-SE/DevGPT. Accessed April 2024.

[18] Stack Overflow. 2023. Stack Overflow Developer Survey 2023. https://survey. stackoverflow.co/2023/ Accessed: 2025-03-24.

[19] OWASP Foundation. 2021. OWASP Top 10: The Ten Most Critical Web Applica- tion Security Risks. https://owasp.org/www-project-top-ten/. Accessed April 2024.

[20] A. Ross, M. Chen, and W. Zaremba. 2023. DevGPT: A dataset for studying developer interactions with language models. GitHub Repository. https://doi.

org/10.1145/3715762 [21] M. L. Siddiq, J. C. Da Silva Santos, R. H. Tanvir, N. Ulfat, F. Al Rifat, and V. Carvalho

Lopes. 2024. Using large language models to generate JUnit tests: An empirical study. In Proceedings of the 28th International Conference on Evaluation and Assess- ment in Software Engineering. 313–322. https://doi.org/10.1145/3661167.3661216

[22] D. Sobania, M. Briesch, C. Hanna, and J. Petke. 2023. An Analysis of the Automatic Bug Fixing Performance of ChatGPT. https://doi.org/10.48550/arXiv.2301.08653 arXiv:2301.10121 [cs.SE]

[23] M. Watanabe, Y. Kashiwa, B. Lin, T. Hirao, K. I. Yamaguchi, and H. Iida. 2024. On the Use of ChatGPT for Code Review. https: //www.researchgate.net/profile/Yutaro-Kashiwa/publication/381513464_ On_the_Use_of_ChatGPT_for_Code_Review_Do_Developers_Like_Reviews_ By_ChatGPT/links/668de8323e0edb1e0fd96a7f/On-the-Use-of-ChatGPT-for- Code-Review-Do-Developers-Like-Reviews-By-ChatGPT.pdf. arXiv:arXiv

[24] C. Wohlin, P. Runeson, M. Höst, M. C. Ohlsson, B. Regnell, and A. Wesslén. 2012. Experimentation in software engineering. Springer Science & Business Media.

[25] Tao Xiao, Hideaki Hata, Christoph Treude, and Kenichi Matsumoto. 2024. De- vGPT: Studying Developer-ChatGPT Conversations. Preprint. Available online at: https://doi.org/10.5281/zenodo.8304091.

[26] B. Yetiştiren, I. Özsoy, M. Ayerdem, and E. Tüzün. 2023. Evaluating the code quality of AI-assisted code generation tools: An empirical study on GitHub Copilot, Amazon CodeWhisperer, and ChatGPT. https://doi.org/10.48550/arXiv. 2304.10778 arXiv:2304.10778 [cs.SE]

13