nodestep.middleware.summarization

SummarizationMiddleware, which replaces older messages with a summary.

See Built-in middleware.

Middleware

SummarizationMiddleware

SummarizationMiddleware(chat, *, trigger, recent, messages_field='messages', token_counter=None, summary_prompt='Summarize the prior conversation concisely. Preserve facts, decisions, and unresolved questions.', summary_message_factory=None, provide_tool=False)

Replace older messages with a summary before a model call once a trigger fires.

Subclass of Middleware.

Only model_node calls are summarized, and not in a node that shares its superstep. The node's stored messages that recent does not keep go to chat; one summary message replaces them in place, with any unstored messages between them, and other unstored messages, such as the system prompt, stay. The node's update stores the summary in place of the first replaced message and removes the others. See Built-in middleware.

Parameters

chat	Chat	Model that writes the summary.
trigger	SummarizationTrigger	When to summarize, checked against the whole request, e.g. MessageCountTrigger(max_messages=40).
recent	RecentMessagesPolicy | RecentTokensPolicy	Which messages to keep verbatim; a tool call and its results are never split.
messages_field	str	State field holding the messages.
token_counter	TokenCounter | None	Counts tokens for the trigger and the recent policy; an ApproximateTokenCounter by default.
summary_prompt	str	Instructions for the summarizer.
summary_message_factory	Callable[[str], BaseMessage] | None	Builds the summary message; a SystemMessage by default.
provide_tool	bool	Return a summarize_context tool from tools(); list it in a node's tools=.

Raises

ModelProviderError	From before_model, if the summarizer returns no text; the model is not called.
GraphConfigError	From before_node, if the state schema does not declare messages_field.

Methods

before_node(ctx)

Remember which messages the node's stored input holds.

before_node(ctx) -> None

Raises

GraphConfigError

If the graph's state schema does not declare messages_field.

async before_model(ctx)

Summarize the request's older stored messages when the trigger fires.

async before_model(ctx) -> Replacement | None

Returns

Replacement or None

The request with the summary, or None to send it unchanged.

Raises

ModelProviderError

If the summarizer returns no text.

after_node(ctx)

Add the summary edits to the node's update.

after_node(ctx) -> Replacement | None

Returns

Replacement or None

The update with the edits, or None when nothing was summarized.

on_error(ctx, error)

Forget the node's pending summary.

on_error(ctx, error) -> None

tools()

Return this instance's summarize_context when provide_tool is set.

tools() -> tuple[Tool, ...]

async summarize_state(state)

Summarize the messages of a state without changing it.

async summarize_state(state) -> SummarizationResult

Parameters

state

Any

Raises

GraphConfigError

If state has no messages_field.

SummarizationResult

SummarizationResult(*, summary, original_message_count, summarized_message_count, kept_message_count, token_count=None, reason)

Result of the summarize_context tool.

Attributes

summary	str	The summary; empty when no older message was left to summarize.
original_message_count	int	Messages before summarizing.
summarized_message_count	int	Older messages the summary replaces.
kept_message_count	int	Recent messages kept as they are.
token_count	int | None	Tokens counted by the trigger; None for MessageCountTrigger.
reason	str	The trigger's reason, or "no_older_messages".

Triggers

SummarizationTrigger

Protocol that decides when to summarize.

Methods

should_summarize(messages, token_counter)

Decide whether the messages should be summarized.

should_summarize(messages, token_counter) -> SummarizationDecision

Parameters

messages	list[BaseMessage]
token_counter	TokenCounter

MessageCountTrigger

MessageCountTrigger(*, max_messages)

Summarize above a number of messages.

Attributes

max_messages

int

Message count above which the trigger fires.

Methods

should_summarize(messages, token_counter)

Check the message count against the limit.

should_summarize(messages, token_counter) -> SummarizationDecision

TokenLimitTrigger

TokenLimitTrigger(*, max_input_tokens, threshold_ratio=0.8)

Summarize above a share of the model's input limit.

Attributes

max_input_tokens	int	The model's input limit.
threshold_ratio	float	Share of the limit above which the trigger fires.

Methods

should_summarize(messages, token_counter)

Check the token count against the limit.

should_summarize(messages, token_counter) -> SummarizationDecision

FixedTokenTrigger

FixedTokenTrigger(*, max_tokens)

Summarize above a fixed number of tokens.

Attributes

max_tokens

int

Token count above which the trigger fires.

Methods

should_summarize(messages, token_counter)

Check the token count against the limit.

should_summarize(messages, token_counter) -> SummarizationDecision

SummarizationDecision

SummarizationDecision(*, should_summarize, reason, token_count=None, message_count)

Whether to summarize, and why.

Attributes

should_summarize	bool
reason	str	The check that was made, such as "token_limit:1200>960".
token_count	int | None	Tokens counted; None for MessageCountTrigger.
message_count	int	Messages checked.

Recent messages

RecentMessagesPolicy

RecentMessagesPolicy(*, keep_recent=20)

Keep the last N messages out of the summary.

Attributes

keep_recent

int

Number of messages kept as they are.

Methods

split(messages, token_counter)

Split messages into those to summarize and those to keep.

split(messages, token_counter) -> tuple[list[BaseMessage], list[BaseMessage]]

Parameters

messages	list[BaseMessage]
token_counter	TokenCounter

Returns

tuple[list, list]

Older and recent messages.

RecentTokensPolicy

RecentTokensPolicy(*, keep_recent_tokens)

Keep the most recent messages within a token budget.

Attributes

keep_recent_tokens

int

Token budget of the messages kept as they are.

Methods

split(messages, token_counter)

Split messages into those to summarize and those to keep.

split(messages, token_counter) -> tuple[list[BaseMessage], list[BaseMessage]]

Parameters

messages	list[BaseMessage]
token_counter	TokenCounter

Returns

tuple[list, list]

Older and recent messages.

Token counting

TokenCounter

Protocol for counting message tokens.

Methods

count_message(message)

Return the token count of one message.

count_message(message) -> int

count_messages(messages)

Return the token count of several messages.

count_messages(messages) -> int

ApproximateTokenCounter

Estimate tokens as characters divided by four.

Methods

count_message(message)

Return the estimated token count of one message.

count_message(message) -> int

count_messages(messages)

Return the estimated token count of several messages.

count_messages(messages) -> int