” Up Level (https://uplevelteam.com) published a 2 page summary of their internal data. We don’t get access to their data and so we can’t tell see if the data support the conclusions.
They measure cycle time of Pull Requests (cough, cough Pull Requests https://agilepainrelief.com/glossary/pull-requests); Defect rates and Overtime (correlated with burnout).
Other sources tell us (some) developers using Copilot have greater work satisfaction.
- Cycle time - decreased by 1.7 minutes
- Bug Rates - +41% increase
- Overtime - both the Copilot users and the Control group reduced their overtime. The Control had a greater reduction: 28% vs 17%
The bug rates are the huge warning sign for me. This correlates with other reports I’ve seen where the mistakes that LLMs make in writing code are often subtle, a missing symbol in an if statement.
So why do people report greater satisfaction using LLMs? My guess is that they’re fun and they make us feel productive even if we pay for that productivity at a later date with messy code and more defects.
Far more important, writing coding is not the bottleneck. We spend more time reading code and understanding the complexity than writing it in the first. As usual, the Theory of Constraints applies here. Work to improve the flow of work at the bottleneck and not elsewhere.
In addition any time we use these tools we need to ask:
- Are we aware of how these things were trained?
- Is the energy consumed a wise use of resources?
https://resources.uplevelteam.com/gen-ai-for-coding
#Influence #Ship30For30