Introduction: The Rise of GenAI in Software Development

So as you all would have witnessed that there has been a lot of shift in how much time would it take for us to code, develop applications, program softwares, create value for ourselves, our teams, our enterprises, our customers.

A lot of Gen-AI code is being created, and most of the big tech would

would confirm that almost 25 to 30% of their code is being generated by GenAI, which is very good because that helps us reduce the technical debt of our engineering teams.

The business can ship more code, more software, more solutions, more products out.

Good for all of us, right?

But then,

Early Warning Signs and Real-World Risks

A cautionary wipe-coding incident

During one of the experiments which we ran six months back, we realized that there are certain lacunas, there are certain drawbacks of using Gen-AI generated code.

So before I dive deeper into the experiment and what we found and how all of you, both from business and technical leadership, could utilize these tools better, I wanted to cite one example.

How many of you wipe code?

Wipe coding?

Loveable, Firebird Studio, Vercel, yes.

So yeah, great thing, right?

It gets us as the mock-up very fast and we can use it to convince our bosses or our collaborators that this is what we like to build.

Have you heard of this incident last month?

Somebody like you and me, they were wipe coding in California on their personal computer and they had their crypto wallet password also available in one of the documents.

And through wipe coding, somebody was able to steal their crypto wallet password and they just lost around 300,000 plus of, yes.

Have you heard of this?

Yeah, I saw it.

You saw it?

Yeah.

Would you like to share something about it?

Because I only knew this much about it.

Oh, see, so that means there are traps which can be misused by the bad actors.

So that's why I would like to bring your attention to this particular piece.

Our Experiment: Measuring Security Degradation

Scope, definitions, and tooling context

So essentially what we did, we ran certain experiments to check what is the degradation of security.

So for the purpose of this presentation, when I say security, it's cybersecurity.

And when I say AI code, it's generative AI-generated code.

When we are using these tools like...

I'll not name the proprietary code generators, but all our big tech and tech companies, AI companies have equivalent of those.

So we could use any of those for this.

And this is my LinkedIn in case anybody would have questions after this.

Feel free to connect, please.

Why developers adopted GenAI—and what we observed

So as we were talking, almost all of us here who develop programs, applications, would have tried using these good solutions, really helpful.

And we have used them initially to generate, subsequently to improve, then refine, and then repeat.

It's been really helpful in the last almost 16 months.

A lot of things have changed at the speed at which we ship products.

But if we pay very peculiar attention to security, you will see there are some observations which we should keep in mind.

So when we explore the current state of AI-generated code security, there is certain current knowledge which we have.

Connectivity to the internet: a double‑edged sword

And if you would remember, March onwards, most of the coding agents got connected to internet.

Internet is a wonderful thing.

You have a lot of information, a lot of knowledge, but it's also full of things which we should not

drive our innovation from, right?

There are a lot of vulnerabilities which can come in through the internet.

And those risks can actually compound all the hard work which we are trying to do by saving some time.

Later on it causes a lot of issues in terms of debugging and

If there are security risk, it causes a lot of issues from the point of view of the security teams, how do you ship that particular code and create a product which is reliable, robust, and wins the trust which you want to win when you're shipping it internally or externally.

And as we use these tools,

These vulnerabilities are available for the bad actors as well to misuse.

It could be at the level of national security, it could be the level of people just doing it for fun, it could be people doing it for seeking ransom.

And as we all know, ransom industry flourishes because they have these tools as much as available as to us to do what they do and it causes a lot of harm.

Key Findings: Iterations Increase Vulnerabilities

So we found just by five iterations of improving your code.

So if you generate code, you will ask whichever tool you're using that please improve it for so and so.

Reduce the number of lines or add some more columns or something else.

1Within five iterations, you will find there will be almost 37.6, almost 40% of increase in critical vulnerabilities of your code.

And that can be really detrimental for the product and also the prestige reputation of your company and the users who use it that can cause a lot of issues.

Study design: baselines, prompts, and iterations

What essentially we did, we designed it in a way that we had 10 baseline samples which were established for the study.

This was just to provide some robust foundation to the findings.

Then we developed four diverse prompting strategies so that it can be applied during code generation.

As you would know, most of the code generation is done through writing proper prompts.

And anybody who uses any of the meta prompt generators to generate your prompts to get best out of your

AI.

Have you heard of meta prompt generators?

Or you write all your prompts yourself?

So again, a piece of advice, if you find that when you prompt any of the language models and you don't get what you want, you can use meta prompt generators like Entropic has something called console, which can you just write in natural language, in simple English or whichever language you want to use, what you want to build.

It will create a meta prompt in almost nine components.

And then you can use that to again prompt

another LLM to create whatever you want to create.

So that way, there are various strategies which you can use.

And in this case, we had four of them.

And then iteration, as I mentioned, we underwent almost 10 iterations for each evaluation.

And it totally allowed us to yield around 400 code samples, which was good enough to build a comprehensive analysis for the purpose of this experiment.

Four prompting strategies we tested

So these were the four prompting strategies.

Efficiency focus, essentially what we did, it was focused on speed of execution.

So a quick code to generate the output and quick iterative cycles.

Then second one was feature focused, where we emphasized on certain functionalities, which was to enhance the capability of the code.

And the third one was focused on security, where we were focusing on certain security vulnerabilities and see how it will impact and how can we use it for mitigation of those vulnerabilities.

And the last one was to improve the ambiguity.

And this was essentially, like I mentioned, we would like the code to be improved because you don't get the best output in the first attempt, so you keep asking.

whichever tool you're using to improve your code.

So how does it specify those outputs?

And if you don't give proper direction, how it changes the output?

Security impact after multiple iterations

So as I mentioned, it was almost 40% increase in vulnerabilities after five iterations.

And this highlights a critical gap where we need very robust monitoring that what's happening, because most of this code is running at the back end, right?

We just leave it.

And now with agentic AI and a lot of solutions which are being built, I mean, you saw yesterday was a Dev Day, and a lot of good stuff has come out of it, Agent Kit.

So a lot of our startup ideas, which were right now pitching for getting some funding, also has got shelved.

But

with those agentic solutions generating code we need some layer of supervision and we'll come to that because if we do not mitigate this this this could be very very uh detrimental to the product and to the company and obviously the users right so this is uh

Vulnerability Landscape Observed

Generally, the distribution of vulnerabilities which were examined, if you see, I'll not dive deeper into it, but this particular presentation and paper is available on archive.

It will be published in IEEE soon, so you can look at it.

But if you see, in terms of vulnerabilities, it ranges from memory safety to almost logical errors, and there are different distribution of it along these strategies.

Memory safety dominates, with crypto and concurrency close behind

Specifically, if we move forward, we'll see that percentage-wise, memory safety stands out as the biggest vulnerability while using these codes.

And this could be something which can impact the performance in the later stages of deployment.

And if you map it along as a heat map, you'll again see that cryptographic and concurrency also has an impact as we move forward.

The Security Paradox of Faster Shipping

So this is what we concluded that though we get very quick solutions using these AI-generated code solutions, but there is a security paradox.

And those new vulnerabilities can actually add a lot of problems for us, and it can be causing a kind of unintended weakness.

though we wanted to actually reduce the technical debt.

But if you ask your CISOs and CIOs and security teams, they have a different take on it, that using these solutions, how robust and how secure the final output would be.

From memory management to growing code complexity

And then we realized that we could perhaps look at it something in terms of memory management evolution, that how things have moved over a period of time, and how we can actually use these solutions to build things a little more secure.

making sure that there are sufficient buffer which is available to map the overflows and incidents.

And as we have talked earlier, there has been a correlation between the code complexity as well as the security vulnerability increase.

As things have moved in last few years in terms of how the complex city has come into AI-generated code, it has reduced the time for us to get the first sample of code, but later on it has caused a lot of vulnerabilities.

Mitigation Strategies and Best Practices

So what could be the mitigating action?

Human-in-the-loop reviews and iteration limits

Perhaps some strategies which we propose as part of this experiment is that we need to have proper mandatory human review.

So I think the whole idea of having a very robust QA team which will check code reviews have to be in place.

We cannot have a crack team of two people doing everything.

I only write the code, I only QA it, I only approve it, and then I just ship it.

It may not be a good thing to do.

And that's why it's required that we have that human to review.

And instead of actually more than four, five times, it's better to limit it to three.

And then perhaps use some of our own human effort to improve it.

Continuous security assessment in production

And it will be important as the code gets deployed, there is continuous analysis of how the security is being, because the risk of security keeps increasing.

It's not only from the generated code.

There are a lot of threats which keep getting into the ecosystem and the environment when the solution is deployed.

And as we have traditionally done for any application which has been put to production, there has to be very regular assessments of vulnerability so that we keep checking what we have shipped.

Because it's all AI-generated, we assume that it is just shipped.

It is not our problem.

Somebody else will look after.

It can be detrimental.

We'll have to keep that in mind.

Organizational Responsibility and Training

If we try to connect this with real-world implication, you will find that the ownership is on all of us.

You could be a business leader.

You could be a tech leader.

You may not be now because you're senior now.

You may not be writing code, but I think the ownership is on all of us to make sure that security gets its due importance as we ship to AI-generated code.

And

There are tools which are available.

There are tools which are being designed to build this particular capability as an inbuilt capability to enhance the security checks for vulnerabilities.

And as an organization, if you see all of us, wherever we work, a large organization or a small startup, everywhere we have to do health and safety training.

We have to do cybersecurity training.

I think it's, again, reiterating on the fact that how important it is, especially in today's day and age.

Ongoing Research and Tooling

What we intend to do is we have been continuing to go ahead with this research.

The same collaborators, two of our papers, which again focus on LLM-based knowledge dilution have been accepted in Europe for two workshops.

This December we'll be presenting it there.

And we intend to actually build these tools and provide it as a platform for everyone to use.

AI alignment, misalignment, and agentic risks

It again brings us back to the concept of AI misalignment.

So anybody who's familiar with alignment of AI or misalignment of AI?

Anybody would like to talk about it just briefly?

Would like to?

Anyone who would like to talk about AI misalignment?

So once the developers or the builders of AI, they have decided that AI will do certain actions, sometimes because of either internal or external misalignment, it starts doing whatever it thinks is the best.

with now agentic systems in place because of their emergent behavior their self-preservation tendencies this misalignment can be more pronounced and that can be a cause of concern so you will see that bad actors may be misusing this particular gap to carry out additional attacks which can cause a lot of issues right and again i think we should not i mean this does not require more emphasis than now then we have to have a human in the loop

and industry will have to adopt it as we move forward.

Resources and how to engage

So you can actually check this on this website also.

We've tried to create a very basic tool again, which is wipe coded just for the initial thing.

You can actually use it as a risk assessment tool.

It'll tell you certain vulnerabilities of your thing.

You can download the paper from archive.

It's available here.

Conclusion and Q&A

we'll continue to publish on this and continue to research and we are open for any discussion which you may have on this so with this I'll stop now if you have any questions please go ahead