Agent Effectiveness
Private PreviewMeasure how much faster your engineering organization ships with Factory, and connect agent spend to the work it produces.
Agent Effectiveness measures how much faster your engineering organization ships with Factory, and connects agent spend to the work it produces.
Most teams that adopt AI coding agents can tell that work is moving faster, but they have a hard time proving it. Survey-based estimates of "how long would this have taken before AI" are unreliable, and existing delivery metrics show how fast work moves without showing how much of that speed came from agents.
Agent Effectiveness answers that question from data your organization already produces: sessions in Factory, and issues, projects, and pull requests in the tools your teams already use.
How it works
Agent Effectiveness reads from your connected project management, issue-tracking, and source control integrations, then links agent activity to the work those tools track. As agent spend is applied, cycle times on projects, issues, and pull requests change measurably, and teams use the recovered time either to take on more work or to raise the bar on the work they already have.
Analysis happens at the organization level and requires no per-repository configuration.
What it measures
ThroughputHow cycle times on projects, issues, and pull requests change as agent spend is applied, and which projects, pods, and users the change is concentrated in.
OutputWhere effort is going, using session intents to classify work and compare the split of engineer time and Factory Standard Credits against your targets.
AttributionThe link between an individual session, the work stream it belongs to, and the issues, projects, and artifacts it produced.
See the Effectiveness dashboard for how each view is presented in the Factory App.
Session intents
A session intent is a classification of what a session was for, such as feature development, maintenance, bug fixing, or exploration. Intents let you compare planned allocation against actual allocation during the quarter rather than after it.
When the mix drifts from your targets, the Output view surfaces the drift while there is still time to act on it.
Attribution and local signals
Attribution connects a session to the concrete work it produced. It relies on local signals, which record the association between a session and the repository work it touched, so an organization-level number can be traced back to specific issues, projects, and artifacts.
Attribution and the Output view only cover the systems you connect. Connect every issue tracker and source control provider your teams use so coverage is complete.
Other usage and adoption metrics
Agent Effectiveness focuses on delivery speed and the link between spend and shipped work. Adjacent measures are documented separately:
| Measure | Where it lives |
|---|---|
| Factory Standard Credits consumption, tool usage, user activity, productivity, and per-user metrics | Analytics API |
| Self-hosted OpenTelemetry metrics export and cost management | Telemetry and analytics |
Requirements
Agent Effectiveness requires organization-level integrations plus the Advanced Analytics enterprise control. See Enable Agent Effectiveness for the full setup.
Agent Effectiveness is in Private Preview. To request access, fill out the form at factory.ai/contact or contact your Factory account team.
Related resources
Connect integrations and turn on Advanced Analytics.
Explore the Throughput, Output, and Attribution views in the Factory App.
Query credits consumption, tool usage, and per-user productivity metrics.
Export metrics to your own observability stack over OpenTelemetry.