Getting AI research agents to produce something useful

One of the more persistent problems while building my investment research desk was getting a valuation workflow to handle incomplete information sensibly. The application could have useful research about a company and still be missing something essential to a per-share valuation, such as a usable diluted share count.

That raised a more useful question than how to force another retry: what should the system save when it can complete part of the work?

The first post walks through the desk itself. This one covers the decisions behind it, including a few things that needed several attempts to get right. I used Codex to help build and debug the implementation, then revised the behavior around what I wanted the application to do.

Make each stage answer a specific question

The research workflow separates discovery, screening, analysis, monitoring, and the decision record. The valuation process needs separation within its own work, too.

The first valuation stage now concentrates on the business: economic assumptions, scenarios, sources, and the inputs needed for calculations. Market-price comparisons and suggested action levels belong later, after that foundation is established.

That keeps the first stage from trying to explain the company, calculate its value, react to the current price, and suggest an action all at once. It also makes the output easier to inspect. I can look at an assumption and ask where it came from before considering what the resulting valuation might mean.

The distinction carries into the implementation. A model can propose and explain inputs, while the application uses explicit calculation code for the supported valuation methods. A market quote doesn't become fair value just because it appeared in a source list.

Source references, dates, asset identity, and scenario details travel with the research. Preserving them makes it possible to compare two versions without quietly treating a fresh quote and an old financial assumption as equally current.

Let incomplete research be useful

Several valuation attempts ran into timeouts or missing evidence. Fixing the request deadlines and output contracts addressed technical failures, but it didn't make unavailable company information appear.

The application can now save a valid partial analysis as a completed research outcome. On the asset, it shows Research available · Valuation pending, with access to the saved analysis and missing inputs. Existing valuations remain available with their original dates and history.

The actual missing-inputs dialog with illustrative research gaps. Missing inputs are explicit; no value is invented to complete the report.

There are limits to how far an agent should improvise. It can explain scenarios using available evidence and identify the additional information needed. It shouldn't quietly substitute a different share-count measure or present an unsupported price target as a completed valuation.

A partial result normally waits for qualifying new company evidence before another automatic attempt. A quote change alone doesn't resolve a missing company input. The wait belongs to that asset, so it doesn't prevent unrelated eligible research from running.

This was an important change in what counted as success. Saving useful work with a clear limitation is a reasonable outcome. Repeatedly asking for the same missing information isn't much of a research strategy.

Make the handoffs explicit

Scout can pass a candidate to Screener, and qualified evidence can lead to Analyst. Monitor can request another analysis when relevant evidence or a recorded assumption changes. Each handoff carries a specific asset and context.

The automatic workflows are bounded. There are limits on new candidates, model-consuming jobs, workflow length, and spending. The worker also checks that automatic execution is still authorized when work resumes. An old queued intent doesn't get permanent permission to run.

Human authority stays separate. An Analyst proposal can't approve its own valuation, replace an accepted thesis, or change a holding. My review and any permitted edit go through their own application commands and history.

Chronicler has a particularly narrow job here. It records decisions from the history that exists. It doesn't fill in a plausible reason I might have had for making them.

The current journal with illustrative entries. My decisions and the Agent journal remain separate views with expandable history.

Keep the website available while work runs elsewhere

The website runs on Vercel. PostgreSQL, provider synchronization, and research workers run on my Mac mini. AWS connects them without exposing a public API or database listener on the Mac.

The basic arrangement is:

• The Mac publishes versioned read snapshots to S3.
• The website's server uses a narrow Lambda boundary to retrieve authorized data and submit permitted commands.
• SQS holds asynchronous commands, which the Mac polls for and processes.
• Completed outcomes are saved and published back for the website to retrieve.

The browser never receives the AWS service credentials. User authentication remains with the website, and the Mac checks command authorization again before applying anything.

This also gives the interface an honest way to handle the Mac being offline. Previously published data can remain available with its freshness information. A new command can remain queued until the worker returns, subject to its expiry. The page must distinguish queued from saved.

Commands have durable identities and recorded outcomes. If a message is delivered twice, the worker can recover the original result rather than applying the change twice. A page refresh can recover the command's status as well.

None of that makes the system instant. It gives a slower or interrupted operation a defined state instead of leaving me to guess whether clicking again will repeat it.

Budget for finishing the job

The recurring research budget has a $100 monthly target and a $110 hard ceiling. The extra room is there for ordinary variation and for work already admitted, with completion headroom reserved when jobs start.

That matters near the monthly limit. Starting a job when very little allowance remains is only useful if the system has accounted for the rest of that job. New admission can stop while previously reserved work finishes within its allowance.

The named automatic workflows also have a $1 lifetime cap apiece, alongside the shared budget and other limits. That's an admission policy, not a claim that every workflow costs a dollar or that the application knows the final provider invoice in advance.

Timeouts need separate treatment. Losing the response doesn't establish that a provider did no work or charged nothing. The ledger keeps an uncertain reservation until the outcome can be reconciled. Historical development costs are recorded separately from the new production budget, while production holds remain visible and counted.

This took some adjustment. A useful budget should control spending without turning a small uncertain charge into a reason to stop every unrelated task.

Test the boundaries as well as the output

The local checks cover permissions, duplicate commands, stale reads, restarts, and missing data. Synthetic fixtures let those checks exercise failures without repeatedly touching private records or buying another model response.

Live checks answer different questions: whether provider access works, results reach the website, and the actual research is worth reading. A passing mock doesn't establish any of those by itself. Conversely, one successful live run doesn't establish that retries or denied writes behave correctly.

The release process also checks encrypted backups and a full database restore. The application uses dedicated database accounts, with normal services kept separate from ownership and migration privileges. That gives the automation a narrower place to operate and preserves a recovery path when a change goes wrong.

The desk is running now, and the remaining work includes better source coverage and evaluating the agents over time. I can check that a report finished, that its sources are attached, and that it respected its permissions. Whether it helped me notice something important still requires reading it.