AI-Generated Code and Security Vulnerabilities: What Every QA Engineer Should Consider

Introduction

AI coding assistants such as GitHub Copilot, ChatGPT, and other code-generation tools are changing how software is developed. Developers can generate functions, APIs, database queries, automated tests, and even complete application components in a matter of seconds.

This improves productivity, but it also introduces an important question:

Just because AI-generated code works does not mean it is secure.

AI models generate code based on patterns learned from their training data and the context provided in a prompt. They can produce useful solutions, but they can also introduce security weaknesses, outdated practices, unsafe dependencies, or incorrect assumptions about how an application should behave.

For development teams, the challenge is no longer just verifying whether the code meets functional requirements. We must also determine whether it protects sensitive data, prevents unauthorized access, and behaves safely under malicious or unexpected conditions.

As QA engineers, we need to expand our testing mindset to include security risks associated with AI-generated code.

1. Why Can AI-Generated Code Introduce Security Vulnerabilities?

AI-generated code is not inherently insecure. However, several factors can make it vulnerable.

1.1 It may follow insecure coding patterns

An AI assistant might generate code that works correctly but uses unsafe input handling, weak validation, or inappropriate security controls.

For example, consider a login API that accepts a user ID from a request and retrieves the corresponding account without verifying whether the authenticated user is authorized to access it.

The API may return the expected response during functional testing. However, an attacker could change the user ID and attempt to access another person's information.

This is an example of a broken access control risk.

1.2 It may make incorrect security assumptions

AI-generated code might assume that:

The frontend has already validated the input.
The user is authorized to access the requested resource.
An API request will always contain valid data.
Error messages do not expose sensitive information.
A third-party library is safe simply because it is widely used.

Security controls must be enforced by the application, particularly on the server side. We should never rely on assumptions that have not been verified.

1.3 It may introduce vulnerable dependencies

AI tools can suggest packages, libraries, or APIs that are outdated, vulnerable, unnecessary, or even incorrectly named.

Installing a dependency without reviewing its origin, maintenance status, permissions, and known vulnerabilities can create additional risks.

1.4 It may expose sensitive information

Developers sometimes share source code, configuration files, logs, or production examples with AI tools to obtain better answers.

If those materials contain API keys, access tokens, passwords, customer information, or proprietary business logic, they may expose sensitive information depending on the tool, its configuration, and the organization's policies.

AI tools should be used in accordance with approved data-handling and privacy policies.

2. Security Vulnerabilities to Look for in AI-Generated Code

When reviewing AI-generated code, I would pay particular attention to the following areas.

Security risk	What to investigate
Injection attacks	Can untrusted input alter SQL queries, shell commands, or other interpreted expressions?
Broken access control	Can users access another user's records or perform unauthorized operations?
Authentication weaknesses	Are identity, session, token, and password-handling mechanisms implemented correctly?
Sensitive data exposure	Are secrets, personal information, or internal details exposed in code, logs, or API responses?
Insecure configuration	Are debug settings, permissive CORS rules, or unnecessary services enabled?
Vulnerable dependencies	Do packages contain known vulnerabilities or come from untrusted sources?
Unsafe file handling	Can uploaded files or user-controlled paths cause unauthorized access or code execution?
Weak cryptography	Are passwords hashed appropriately and established cryptographic libraries used correctly?
Insufficient error handling	Do exceptions reveal database details, internal paths, tokens, or stack traces?
Business logic vulnerabilities	Can users bypass payment, approval, policy, or transaction rules?

These risks align with established application-security guidance, including the OWASP Top 10. The appropriate checks depend on the application architecture, data sensitivity, and threat model.

3. How Should QA Engineers Approach AI-Generated Code?

Traditional testing often begins with the question: "Does the feature work as expected?"

Security testing adds another question: "How could someone misuse this feature?"

Consider an insurance application that allows customers to retrieve their policy details.

A functional test might verify that a customer can retrieve an existing policy using a valid policy ID.

A security-focused test should also investigate:

Can Customer A retrieve Customer B's policy by changing the policy ID?
Can an unauthenticated user access the endpoint?
Can a user with a restricted role access administrative information?
Does the API disclose personal information in error responses?
Can malformed input cause an exception or unintended database query?
Are access-control checks applied consistently across related endpoints?

This is particularly important for applications that process customer data, financial transactions, insurance claims, or payments.

The objective is to validate both intended functionality and resistance to misuse.

4. Practical Measures to Reduce Security Risks

Measure 1: Review AI-generated code before merging

AI-generated code should follow the same review process as manually written code.

During code review, investigate:

Whether input validation is appropriate.
Whether authentication and authorization are enforced on the server.
Whether sensitive data is handled safely.
Whether exceptions and logs reveal confidential information.
Whether the implementation follows established security patterns.
Whether unnecessary permissions or dependencies have been introduced.

Avoid approving code simply because it is readable, passes unit tests, or was generated by a well-known AI tool.

Measure 2: Use automated security scanning in CI/CD

Security checks should be integrated into the development pipeline instead of being postponed until the end of a release.

Useful categories of security tools include:

SAST — Static Application Security Testing: Examines source code for potentially vulnerable patterns.
SCA — Software Composition Analysis: Identifies vulnerable or risky third-party dependencies.
DAST — Dynamic Application Security Testing: Tests a running application for exploitable weaknesses.
Secret scanning: Detects accidentally committed credentials and tokens.
Infrastructure-as-code scanning: Checks cloud and deployment configuration for security problems.

For example, a GitHub Actions pipeline could run unit tests, dependency checks, static analysis, and secret scanning before allowing a pull request to be merged.

Security gates should be configured to block releases when agreed-upon critical findings remain unresolved, while avoiding unnecessary failures caused by untriaged false positives.

Measure 3: Design security-focused test cases

QA engineers should deliberately include negative, abuse-case, and authorization tests.

For an API, this could include:

Missing or invalid authentication tokens.
Expired tokens.
Requests with insufficient permissions.
Attempts to access another user's resources.
Unexpected data types and boundary values.
Malformed or excessively large requests.
Attempts to bypass business rules.
Repeated requests that could trigger abuse or resource exhaustion.

Testing should use authorized test environments and controlled test data.

Measure 4: Validate third-party dependencies

Before introducing a new package suggested by an AI assistant, check:

Whether the package actually exists and is published by a trustworthy source.
Whether it is maintained and compatible with the project.
Whether it has known vulnerabilities.
Whether its permissions and functionality are necessary.
Whether the project has approved alternatives.

Use dependency lockfiles, automated vulnerability alerts, and an approved dependency policy to make this process repeatable.

Measure 5: Protect secrets and sensitive data

Never assume that generated configuration is safe for production.

Check that:

API keys and passwords are not hardcoded.
Secrets are stored in approved secret-management systems.
Sensitive values are not committed to source control.
Logs do not expose authentication tokens or personal data.
AI prompts and uploaded files comply with organizational data-handling rules.

If a real credential is exposed, removing it from the code is not enough. Revoke or rotate it and investigate whether it was accessed.

Measure 6: Use AI as a security assistant, not the final authority

AI can help identify suspicious code, suggest secure alternatives, and generate negative test cases.

For example, a QA engineer could ask an approved AI assistant:

"Review this API implementation for possible broken access control, injection, sensitive data exposure, and insufficient input validation. Identify the relevant code paths and suggest security tests."

However, AI-generated security reviews can miss vulnerabilities or report problems that do not exist. Validate its findings through code inspection, established security tools, and testing.

AI should support security expertise rather than replace it.

5. Example: Testing an AI-Generated API

Imagine that an AI assistant generates an endpoint that retrieves an insurance policy.

A basic functional test might look like this:

it('retrieves an existing policy', () => {
  cy.request({
    method: 'GET',
    url: '/api/policies/POLICY-1001',
    headers: {
      Authorization: `Bearer ${customerToken}`
    }
  }).then((response) => {
    expect(response.status).to.eq(200);
    expect(response.body.policyId)
      .to.eq('POLICY-1001');
  });
});

This verifies that the expected policy can be retrieved. However, it does not prove that the endpoint is secure.

Additional security tests should verify authorization boundaries:

it('prevents access to another customer policy', () => {
  cy.request({
    method: 'GET',
    url: '/api/policies/POLICY-2002',
    headers: {
      Authorization: `Bearer ${customerToken}`
    },
    failOnStatusCode: false
  }).then((response) => {
    expect([403, 404]).to.include(response.status);
  });
});

Here, the example assumes that customerToken belongs to a customer who is not authorized to access POLICY-2002, and that the application's documented security policy uses either 403 or 404 for this situation.

The test should use valid test accounts and policies. The expected status code must match the application's actual security contract.

This is a simple illustration of how security requirements can be converted into automated API tests. The same principle applies to UI tests, integration tests, and end-to-end workflows.

Importantly, a passing test does not establish that an endpoint is fully secure. The team must also examine other authorization paths, response data, and potential attack techniques.

6. Should AI-Generated Code Have a Different Definition of Done?

I believe this is an important discussion for Agile teams.

AI-generated code should not necessarily have a completely separate Definition of Done. Instead, the existing Definition of Done should explicitly cover security requirements that apply to all code, regardless of how it was written.

For example, a Definition of Done could require:

Functional acceptance criteria are satisfied.
Relevant unit, integration, and regression tests pass.
Authorization and input-validation requirements are verified.
Required static analysis and dependency checks are completed.
No unresolved security findings exceed the team's agreed risk threshold.
Secrets and sensitive information are handled according to policy.
Code review is completed.
Security findings are documented and remediated or formally accepted by an authorized risk owner.

The specific requirements should reflect the application's risk level. A customer-facing payment service may need stronger controls than a small internal reporting utility.

The important principle is consistency: code generated by AI must meet the same security standards as code written by a developer.

7. A Practical Security Checklist for QA Teams

Before approving a feature that contains AI-generated code, ask the following questions:

Has the generated code been reviewed by a developer?
Are authentication and authorization enforced correctly?
Are untrusted inputs validated and safely handled?
Have negative and abuse-case tests been considered?
Are sensitive data and credentials protected?
Have dependencies been checked for known vulnerabilities?
Have applicable SAST, SCA, secret-scanning, and DAST checks been completed?
Are error messages and logs free from sensitive information?
Have security-relevant business rules been tested?
Have identified vulnerabilities been fixed or formally risk-accepted?
Is the code compliant with the organization's approved AI usage policy?

Not every tool needs to run against every change. The team should establish risk-based rules for which checks run on each pull request, on scheduled builds, and before production releases.

Conclusion

AI-generated code offers significant opportunities to improve developer productivity and accelerate software delivery. However, faster code generation must not lead to weaker security practices.

The real risk is not simply that AI can generate vulnerable code. It is that teams may trust generated code too quickly because it looks correct and passes functional tests.

For QA engineers, this creates an opportunity to strengthen the role of quality engineering. We must go beyond verifying expected behavior and investigate unauthorized access, malicious inputs, sensitive data exposure, vulnerable dependencies, and unintended business outcomes.

The best approach is to combine human code review, security-focused test design, automated scanning, dependency management, and risk-based release controls.

Ultimately, AI can help us write code faster, but security must remain a shared responsibility across development, QA, security, and DevOps.

The goal is not to avoid AI-generated code. It is to use it responsibly and ensure that every line of code, regardless of its origin, earns our trust through verification.

Shavanthas Blog

Search This Blog