PaperlayerView exact revision on arXiv

Source-linked AI summary

Oculi: A Conversational Agentic Platform for Automated Credit Risk Analysis

Vennise Ho, Kristian Diana, Sandy Mourad, Milena Pilipovic, Vineel Nagisetty, Hossein Hajimirsadeghi

arXiv:2608.28944v1cs.AIcs.LG

TL;DR

Credit-risk analysis is constrained by manual, fragmented workflows that limit exploration to familiar segments. Oculi combines a conversational three-layer architecture with deterministic statistical analysis and LLM-guided feature selection, achieving 63% coverage of material bivariate segments in its mortgage-portfolio evaluation. The evaluation is limited by simulated rather than live LLM guidance and by a nine-feature search space that is smaller than production portfolios.

  • Problem

    Manual credit-risk workflows limit exploration and leave analysts repeatedly examining familiar segments.

  • Method

    Oculi combines a three-layer conversational architecture with deterministic statistical methods and LLM-guided feature selection for risk-segment discovery.

  • Results

    63% coverage of material bivariate segments was achieved against constrained brute-force enumeration.

  • Takeaways & Limitations

    Oculi supports conversational querying, statistical testing, visualization, and novel risk-segment discovery through chat.

  • Takeaways & Limitations

    The evaluation uses a simulated context-sensitive heuristic and only 9 features, so live-LLM behavior and production-scale coverage remain uncharacterized.

Abstract

from arXiv · show

Credit risk analysis in financial institutions traditionally requires analysts to manually write SQL queries, run statistical computations, and build visualization dashboards. This is a time-consuming workflow that limits exploration to familiar segments. We introduce \textbf{Oculi}, a conversational platform that transforms natural language questions into comprehensive credit risk analyses, complete with data queries, statistical testing, and interactive visualizations. Oculi employs a three-layer architecture that separates reasoning (LLM-powered agent), execution (Model Context Protocol tool servers), and presentation (agentic UI), enabling analysts to discover high-risk portfolio segments. Within Oculi, a new segment discovery pipeline is proposed that combines deterministic statistical methods with LLM-guided feature selection, leveraging LLM semantic domain knowledge alongside data-driven metrics to identify meaningful, actionable portfolio segments. Evaluated on a mortgage portfolio with 200+ features, Oculi demonstrates effectiveness in discovering material risk segments previously intractable through manual exploration, reducing time-to-insight significantly while maintaining auditability and statistical rigor.

1 Introduction

Oculi addresses fragmented, reactive credit-risk workflows with a conversational platform that combines agentic reasoning, tool execution, interactive presentation, and segment discovery. Its evaluation reports material-segment coverage while reducing the combinations explored.

  • Traditional analysis requires SQL querying, statistical analysis, and manual dashboard construction, leading analysts to repeatedly examine only 10–15 familiar segments.
  • Manual workflows face exploration bottlenecks, fragmented context, and reactive analysis across hundreds of portfolio features.
  • Existing AI analytics systems generally lack the statistical rigor and domain-specific capabilities required for production financial analysis.
  • Oculi separates reasoning, execution, and presentation into an LLM agent, MCP tool servers, and generative UI while maintaining auditability.
  • Its segment-discovery pipeline combines deterministic statistical methods with LLM-guided feature selection to identify statistically significant, material risk segments without combinatorial explosion.
  • 63% coverage of material bivariate segments required evaluating 74% of the combinations used by constrained brute-force enumeration.

2 Related Work

Related systems support natural-language querying, visualization, agentic tool use, or subgroup discovery, but Oculi combines these capabilities with statistical rigor and credit-risk-specific exploration. The paper positions this combination as extending beyond query translation toward autonomous analytical exploration.

  • Natural-language analytics systems translate questions into queries or visualizations but generally lack multi-step reasoning and statistical analysis for risk assessment.
  • Agentic frameworks provide tool-augmented reasoning, while Oculi adds domain-specific constraints, statistical rigor, and persistence requirements.
  • Traditional feature-selection and subgroup-discovery methods do not address high-dimensional, high-cardinality credit-risk exploration with materiality and seasonal-adjustment requirements.
  • Conversational BI tools provide dashboard interfaces but require pre-built data models and do not autonomously chain analytical operations or discover novel segments.
  • Oculi combines conversational interaction, agentic reasoning, rigorous statistical analysis, and domain-specific credit-risk deployment.

3 Problem Formulation

The problem formulation defines portfolio segments as conjunctions of feature-value filters and seeks segments that are statistically significant, materially sized, and actionable. The search is combinatorial, making exhaustive enumeration impractical at production scale.

  • A segment is defined by a conjunction of feature-value filters, with depth k equal to the number of filters.
  • For each segment, the framework computes risk metrics and their period-over-period changes.
  • Candidate material segments must satisfy statistical significance, materiality, and actionability criteria.
  • Bivariate segment counts grow as O(d^2 · c^2), creating a combinatorial search problem as feature count and cardinality increase.
  • 318,000 candidate segments arise at depth 2 and approximately 84,000,000 at depth 3 in the stated production-scale example.
  • Brute-force enumeration becomes computationally infeasible at production scale, while manual analysis explores fewer than 20 segments per month.

4 System Architecture

Oculi uses a layered architecture in which a ReAct agent reasons and orchestrates tools, MCP servers execute data and analytical operations, and a generative UI presents results. Its segment-discovery pipeline begins with deterministic scanning and filtering before LLM-guided deepening.

  • 4 System Architecture: The frontend, agent, and MCP servers respectively handle interaction and presentation, reasoning and orchestration, and deterministic data and analysis operations.
  • 4.1 Frontend: Agentic UI: The generative UI renders analysis cards, interactive charts, and comparison grids directly in the chat stream.
  • 4.1 Frontend: Agentic UI: The agent streams text, tool invocations, and UI triggers while retaining shared analytical state for context-aware responses.
  • 4.1 Frontend: Agentic UI: Chart specifications are stored separately and retrieved by short identifiers, keeping large visualization JSON out of the LLM reasoning context.
  • 4.2 Agent: ReAct Loop with LangGraph: The LangGraph ReAct agent repeatedly prompts the LLM, executes returned tool calls, updates memory with results, and returns a final response when no further calls are needed.
  • 4.3 MCP Servers: Tool Execution Layer: MCP exposes discoverable, schema-described tools over HTTP, allowing independent analytical services and multiple servers to be orchestrated without modifying the agent.
  • Segment Discovery Pipeline: The discovery pipeline takes portfolio, features, metric, period, and thresholds as input and produces ranked segments through univariate scanning, materiality filtering, and LLM-guided deepening.

5 Segment Discovery Pipeline

Oculi’s segment discovery pipeline combines deterministic statistical screening with LLM-guided feature exploration and an optional decision-tree path. It cross-references discoveries, ranks material segments, and preserves auditability through deterministic computation.

  • The pipeline discovers material, statistically significant risk segments without brute-force enumeration by separating deterministic computation from LLM-guided exploration.
  • Deterministic screening: Univariate scanning generates feature-value segments, after which materiality thresholds require sufficient sample size.The supplied thresholds include account_count(S) > nmin.
  • LLM-guided deepening: For each material univariate segment, the LLM receives segment context and available feature descriptions, then suggests 1–5 features for multivariate exploration.Suggestions include reasoning about regional variation, delinquency changes, and leverage-related risk.
  • LLM-guided deepening: Valid feature picks are checked, hallucinated names are dropped, and multivariate segments are computed deterministically with the materiality filter reapplied.The validation also caps selections at five picks and provides a fallback for malformed responses.
  • Dual discovery: The optional decision tree finds interaction effects through MSE-based splits, while the LLM contributes domain knowledge that the tree cannot learn from a single noisy period.Leaves and internal nodes define segments, and feature importances provide a global ranking of variance in metric movement.
  • Cross-reference and ranking: Results from both paths are matched, labeled as LLM-only or tree-only when appropriate, deduplicated, and ranked by contribution ϕ(Si).Segments found by both methods retain the LLM metrics and receive the tree path explanation for interpretability; ϕ balances segment size and deviation magnitude.

6 Conversational Workflow Example

Oculi turns a natural-language risk question into an orchestrated analysis that returns segment statistics, explanations, and interactive visualizations. Follow-up questions preserve the established analytical context.

  • An analyst can ask which mortgage segments are riskiest, prompting the agent to parse intent and call the segment-exploration tool.
  • The MCP service returns segment breakdowns with z-scores and contributions, which the agent synthesizes into an explanation.
  • The frontend streams a text summary, creates an analysis card, and renders a Vega-Lite chart from the segment data.
  • Follow-up questions such as drilling into Alberta or comparing with last quarter retain the full conversation context.

7 System Implementation

Oculi is implemented as three independently deployed services for the frontend, agent, and MCP server. Authenticated communication connects the services, while the MCP layer provides tool routing and shared persistence.

  • Frontend: The proof-of-concept uses a Next.js and React frontend with CopilotKit, Tailwind CSS, and an enterprise design system.
  • Agent: The agent runs FastAPI and LangGraph on Python 3.11, deployed in Docker on OpenShift.
  • MCP server and data: The MCP server uses FastMCP, DuckDB for S3 Parquet queries, and MongoDB for persistence.
  • Authentication and model access: The system uses Cohere Command-A through an enterprise OAuth-authenticated LLM gateway and integrates enterprise SSO.
  • Deployment: Frontend, agent, and MCP server containers run independently on ports 3000, 8080, and 8081, respectively.The agent container includes the MCP gateway, while the MCP server co-locates business logic with tool definitions.
  • Communication: Inter-service communication is authenticated, with AG-UI streaming, Streamable HTTP JSON-RPC, and MongoDB shared persistence.

8 Empirical Evaluation

Oculi’s empirical evaluation uses an anonymized mortgage portfolio to compare segment-discovery strategies, feature-suggestion validation, and pipeline latency. The evaluation reports stronger coverage from context-sensitive LLM-guided selection, robust filtering of invalid suggestions, and an estimated 10–15-second production latency with parallelization, while noting important scope limitations.

  • Evaluation Setup: The evaluation uses an anonymized mortgage portfolio with 9 features, 34 univariate segments, and a month-over-month delinquency metric from March to April 2024.The portfolio contains approximately 2.4 million accounts; delinquency increased by 8.3 bps, from 45.2 to 53.5 bps.
  • Evaluation Setup: Constrained brute-force enumeration identifies 93 material bivariate segments from 220 evaluated segments and serves as the reference set.The brute-force reference is tractable for the 9-feature evaluation but not for production-scale portfolios.
  • Discovery Quality: LLM-guided selection achieves 63% coverage while evaluating 162 of 220 combinations, or 74% of the constrained brute-force search.It outperforms the fixed-priority deterministic heuristic by 22 percentage points, with 63% versus 41% coverage.
  • Discovery Quality: The decision-tree baseline finds 2 material segments, achieving 2% coverage while using only 3 of the 9 available features.The evaluation attributes this result to shallow greedy selection on data where delinquency shifts are distributed across many features.
  • Feature Suggestion Accuracy: Across 20 simulated LLM responses, validation silently drops all 10 hallucinated features, representing 23.8%, without affecting pipeline execution.The validation layer handles malformed JSON, freeform text, mixed valid and invalid feature names, and missing reasoning fields.
  • Feature Suggestion Accuracy: The 32 valid suggestions cover 8 of 9 available features, or 89%, indicating broad feature exploration.The evaluation also reports that bivariate segment computation is the dominant cost at 60%.
  • Latency Profile: With parallelization and LLM round-trips, estimated production latency is 10–15 seconds end-to-end.LLM feature suggestions take 1–3 seconds per round-trip, while DuckDB queries parallelize well across segment evaluations.
  • Limitations: The evaluation’s main limitation is that LLM-guided selection uses a simulated context-sensitive heuristic rather than live LLM inference.The authors also note that the 9-feature setting is tractable, whereas 200-plus-feature production portfolios may amplify priority-cutoff effects.

9 Discussion

Oculi separates reasoning, execution, and presentation to support auditable financial analytics, while combining LLM-guided exploration with deterministic methods for risk-segment discovery. The discussion identifies current limits in depth, scale, temporal and causal analysis, feedback, multi-metric discovery, and tree-based coverage, alongside proposed extensions.

  • Architecture: The three-layer architecture separates reasoning, execution, and presentation while supporting modularity, scalability, auditability, and safety.Tool calls are logged, deterministic stages are reproducible, and operations are mediated through schema-validated MCP calls.
  • Architecture: The LLM handles conversational orchestration and feature selection, while MCP tools perform all queries and statistical calculations deterministically.The LLM’s role is strategic rather than computational.
  • Limitations: The current pipeline performs one LLM-guided deepening pass at depth 2, while production-scale feature spaces create rapidly growing candidate sets.The evaluation uses 9 features and 220 brute-force candidates, whereas 200+ production features yield approximately 318,000 depth-2 candidates and approximately 84 million depth-3 candidates.
  • Limitations: Current scope excludes proactive scheduling, multi-period trends, causal attribution, analyst-feedback personalization, and joint multi-metric anomaly discovery.These capabilities require infrastructure, temporal reasoning, causal inference, personalization, or multivariate scoring extensions.
  • Discovery: The dual discovery paths combine decision-tree variance partitioning with LLM domain reasoning to identify complementary signals and blind spots.Segments found by both methods are treated as stronger signals, while one-path discoveries reveal coverage gaps.
  • Generality: The architecture and pipeline are positioned for domains with high-dimensional structured data, segment analysis, domain experts with limited programming skills, and auditability requirements.Examples include fraud detection, churn, healthcare outcomes, supply-chain risk, and clinical-trial subgroup analysis.

10 Conclusion

Oculi is a conversational credit-risk platform that turns natural-language questions into analytical workflows through separated reasoning, execution, and presentation layers. Its deterministic and LLM-guided segment-discovery pipeline achieved 63% coverage of material bivariate segments in an anonymized mortgage evaluation, while production-scale search remains computationally infeasible for brute-force enumeration.

  • Conclusion: Oculi transforms natural-language questions into credit-risk workflows for querying data, running statistical tests, generating visualizations, and discovering risk segments.These capabilities are delivered through a chat interface.
  • Conclusion: The segment-discovery pipeline combines deterministic statistical methods with LLM-guided feature selection while keeping computation deterministic for auditability.The LLM makes strategic feature-exploration decisions rather than performing numerical operations.
  • Evaluation: 63% coverage of material bivariate segments was achieved against constrained brute-force enumeration on a real anonymized mortgage portfolio.The 220-segment reference set was tractable because evaluation was constrained to 9 features.
  • Implications: At production scale, approximately 318,000 bivariate candidates make brute-force enumeration computationally infeasible, increasing the importance of intelligent feature selection.The production setting described contains 200+ features.
  • Implications: Oculi is presented as augmenting expert analysts by automating routine workflows, scaling exploratory analysis, and surfacing insights manual exploration may miss.Future work includes deeper segments, temporal trends, proactive pre-computation, and analyst-feedback learning.
Loading 2608.28944v1…