Skip to content

fix: normalize OpenAI/Azure cached input tokens for correct cost persistence#23210

Closed
ibetitsmike wants to merge 1 commit intomainfrom
mike/cache-tokens-wtzm
Closed

fix: normalize OpenAI/Azure cached input tokens for correct cost persistence#23210
ibetitsmike wants to merge 1 commit intomainfrom
mike/cache-tokens-wtzm

Conversation

@ibetitsmike
Copy link
Collaborator

@ibetitsmike ibetitsmike commented Mar 18, 2026

Problem

OpenAI reports prompt_tokens (mapped to InputTokens) including cached tokens, and separately reports cached_tokens (mapped to CacheReadTokens). The persistence path stored both values as-is, causing:

  • Double-counted costs: CalculateTotalCostMicros priced InputTokens at input rate + CacheReadTokens at cache rate, billing the cached portion twice.
  • Inflated input token counts in the model usage summary table.

Anthropic reports these fields separately (InputTokens = non-cached only), so only OpenAI/Azure are affected.

Fix

  • Normalization helper (coderd/chatd/usage.go): normalizePersistedUsage(provider, raw) subtracts CacheReadTokens from InputTokens for openai and azure providers. Returns a copy — raw step.Usage stays untouched for compaction/context accounting.
  • Persistence path (coderd/chatd/chatd.go): Uses normalized usage for both usageForCost and InsertChatMessage params.
  • Migration 000444: Corrects historical input_tokens and recomputes total_cost_micros for affected OpenAI/Azure rows. Preserves NULL (unpriced) costs.
  • SDK docs (codersdk/chats.go): Clarifies InputTokens is the non-cached billable bucket; TotalTokens is raw provider-reported.

Tests

  • TestNormalizePersistedUsage — 6 table cases (subtract, clamp, explicit zero, both-zero, Anthropic unchanged, no-cache)
  • TestPersistStepNormalizesCachedInput — end-to-end integration with mock OpenAI server
  • TestContextTokensFromUsage — compaction still uses raw (non-normalized) usage
  • TestCalculateTotalCostMicros — Anthropic separate-bucket regression
  • TestChatCostSummary — cache tokens in summary aggregation
  • Migration TestMigrate (including UpDownUp) passes

chatgpt-codex-connector[bot]
chatgpt-codex-connector bot reviewed Mar 18, 2026
View reviewed changes
Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7f98e9acaa

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ibetitsmike ibetitsmike force-pushed the mike/cache-tokens-wtzm branch from 7f98e9a to 3d0aee4 Compare March 18, 2026 13:31
@ibetitsmike
Copy link
Collaborator Author

ibetitsmike commented Mar 18, 2026

@codex review

chatgpt-codex-connector[bot]
chatgpt-codex-connector bot reviewed Mar 18, 2026
View reviewed changes
Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3d0aee421b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

kylecarbs
kylecarbs reviewed Mar 18, 2026
View reviewed changes
@ibetitsmike
Copy link
Collaborator Author

ibetitsmike commented Mar 18, 2026

@codex review

chatgpt-codex-connector[bot]
chatgpt-codex-connector bot reviewed Mar 18, 2026
View reviewed changes
Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 04738d1ed6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ibetitsmike
Copy link
Collaborator Author

ibetitsmike commented Mar 18, 2026

@codex review

chatgpt-codex-connector[bot]
chatgpt-codex-connector bot reviewed Mar 18, 2026
View reviewed changes
Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0c2d5e4f7a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ibetitsmike
Copy link
Collaborator Author

ibetitsmike commented Mar 18, 2026

@codex review

chatgpt-codex-connector[bot]
chatgpt-codex-connector bot reviewed Mar 18, 2026
View reviewed changes
Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 949eeb5759

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ibetitsmike ibetitsmike force-pushed the mike/cache-tokens-wtzm branch from 949eeb5 to a5a7514 Compare March 18, 2026 14:55
@ibetitsmike
Copy link
Collaborator Author

ibetitsmike commented Mar 18, 2026

@codex review

chatgpt-codex-connector[bot]
chatgpt-codex-connector bot reviewed Mar 18, 2026
View reviewed changes
Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a5a75147f6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ibetitsmike
Copy link
Collaborator Author

ibetitsmike commented Mar 18, 2026

@codex review

chatgpt-codex-connector[bot]
chatgpt-codex-connector bot reviewed Mar 18, 2026
View reviewed changes
Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2e374c7b3d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ibetitsmike ibetitsmike force-pushed the mike/cache-tokens-wtzm branch from 2e374c7 to 74543a1 Compare March 18, 2026 15:31
@ibetitsmike
Copy link
Collaborator Author

ibetitsmike commented Mar 18, 2026

@codex review

chatgpt-codex-connector[bot]
chatgpt-codex-connector bot reviewed Mar 18, 2026
View reviewed changes
Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 74543a19bf

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ibetitsmike
fix: normalize OpenAI/Azure cached input tokens for correct cost pers…
3e05ad2 
…istence

OpenAI reports prompt_tokens including cached tokens, and separately
reports cached_tokens. The persistence path stored both values as-is,
causing double-counted costs and inflated input token counts.

Changes:
- Add normalizePersistedUsage() helper that subtracts CacheReadTokens
  from InputTokens for openai/azure providers before persistence
- Apply normalized usage for both cost calculation and DB insert
- Add migration 000445 to correct historical rows (delta-only cost
  adjustment to preserve other pricing components)
- Extend test fixtures, add unit/integration/regression tests
- Clarify ChatMessageUsage field semantics in SDK docs
@ibetitsmike ibetitsmike force-pushed the mike/cache-tokens-wtzm branch from 74543a1 to 3e05ad2 Compare March 18, 2026 15:48
@ibetitsmike
Copy link
Collaborator Author

ibetitsmike commented Mar 18, 2026

@codex review

@chatgpt-codex-connector
Copy link

chatgpt-codex-connector bot commented Mar 18, 2026

Codex Review: Didn't find any major issues. Swish!

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ibetitsmike
Copy link
Collaborator Author

ibetitsmike commented Mar 18, 2026

Fortunately will be fixed upstream.

@github-actions github-actions bot locked and limited conversation to collaborators Mar 18, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@chatgpt-codex-connector chatgpt-codex-connector[bot] chatgpt-codex-connector[bot] left review comments

@kylecarbs kylecarbs kylecarbs approved these changes

Assignees

@ibetitsmike ibetitsmike

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@ibetitsmike @kylecarbs