I don’t really understand using AI for code reviews. Shouldn’t it be the other way around?
I have my setup hooked up to SonarQube and compell it to run all its changes through that. I prefer a static analysis tool with predefined checks to keep the agent in line. I just have it do a TDD workflow using that as part of the refactor step.
My experience is that prepping a code review with LLM is a safer bet than having a person review LLM code. None of this should replace static analysis or human code review, but it can provide a rapid feedback about something that looks off either to the original developer or helping a reviewer catch a detail that might have escaped their attention span. LLM missteps don’t matter so much, since the audience is a human (an exceedingly high false positive rate is annoying, but not a disastor). Also why it makes for a handy facility for security, mistakes don’t matter, missing something is no worse than before, false positives can be ignored.
If your scenario is well served by TDD with absoutely comprehensive test cases that can grade with absolute perfect coverage of all the scenarios, then code gen can thrash on that and make progress, though it’s a bit token heavy because there’s a fair amount of guesses and reguessing. But if you AI gen the tests and the code, then you might get into weird states. Even if you didn’t mean to have the AI mess with the tests things can go south. A couple weeks back I tried to prompt up a change that I thought should have been reasonably within the capabilities of the model, along the way it failed a long standing test and fixed it by changing the test, making the determination that the test case needed correcting to let it pass, instead of trying to pass the test case. The test case was, in fact, correct, it screwed up.
Part of the problem is ‘writing code’ is about as diverse as ‘writing text’, and some folks are working in territory that the LLMs can chew through competently and others are places where the LLMs struggle, and then there’s the person lulled into a sense of security by being in the former category until one day suddenly they are in the latter unexpectedly for no apparent reason.
My current company requires human code reviews as well but, on commit, it gets reviewed by copilot. Codepilot tends to make some mistakes, but it does catch some things (and, yes, I’d rather be using non-ai tools that also do that, but not my choice).
I don’t really understand using AI for code reviews. Shouldn’t it be the other way around?
I have my setup hooked up to SonarQube and compell it to run all its changes through that. I prefer a static analysis tool with predefined checks to keep the agent in line. I just have it do a TDD workflow using that as part of the refactor step.
My experience is that prepping a code review with LLM is a safer bet than having a person review LLM code. None of this should replace static analysis or human code review, but it can provide a rapid feedback about something that looks off either to the original developer or helping a reviewer catch a detail that might have escaped their attention span. LLM missteps don’t matter so much, since the audience is a human (an exceedingly high false positive rate is annoying, but not a disastor). Also why it makes for a handy facility for security, mistakes don’t matter, missing something is no worse than before, false positives can be ignored.
If your scenario is well served by TDD with absoutely comprehensive test cases that can grade with absolute perfect coverage of all the scenarios, then code gen can thrash on that and make progress, though it’s a bit token heavy because there’s a fair amount of guesses and reguessing. But if you AI gen the tests and the code, then you might get into weird states. Even if you didn’t mean to have the AI mess with the tests things can go south. A couple weeks back I tried to prompt up a change that I thought should have been reasonably within the capabilities of the model, along the way it failed a long standing test and fixed it by changing the test, making the determination that the test case needed correcting to let it pass, instead of trying to pass the test case. The test case was, in fact, correct, it screwed up.
Part of the problem is ‘writing code’ is about as diverse as ‘writing text’, and some folks are working in territory that the LLMs can chew through competently and others are places where the LLMs struggle, and then there’s the person lulled into a sense of security by being in the former category until one day suddenly they are in the latter unexpectedly for no apparent reason.
My current company requires human code reviews as well but, on commit, it gets reviewed by copilot. Codepilot tends to make some mistakes, but it does catch some things (and, yes, I’d rather be using non-ai tools that also do that, but not my choice).