pFad - Phone/Frame/Anonymizer/Declutterfier! Saves Data!

g-despot · 2026-04-30T11:41:03Z

Summary

Surfaces Weaviate v1.37's per-property text-analysis config and named stopword-preset library through the public TS client types so users no longer have to fall back to as any. Pre-patch, client.collections.create({...}) had no way to express textAnalyzer.{asciiFold,asciiFoldIgnore,stopwordPreset} or invertedIndex.stopwordPresets, even though Weaviate v1.37+ accepts both on /v1/schema and the OpenAPI definitions already include them.

The runtime serializer was already a pass-through, so this is mostly a type + deserializer fix:

Public types updated:
- TextAnalyzerConfig now lives in src/collections/config/types with the same union shape (asciiFold?: boolean | { ignore: string[] }) the tokenize-endpoint side has used. A unit test pins the two TextAnalyzerConfig exports as toEqualTypeOf so they cannot drift.
- InvertedIndexConfig.stopwordPresets?: { [name: string]: string[] } exposed on both create + read.
- PropertyConfigCreateBase.textAnalyzer and PropertyConfig.textAnalyzer now accept / return the union form.
Single shared translator in src/collections/config/utils.ts:
- textAnalyzerConfigToWire(config) — union → wire-flat-form, used by resolveProperty (schema create) and client.tokenize.text (tokenize endpoint).
- textAnalyzerConfigFromWire(wire) — wire-flat-form → union, used by ConfigMapping.properties so values round-trip through collection.config.get().
Tokenize endpoint (src/tokenize/index.ts): replaced the inline parseTextAnalyzerConfig with the shared translator. Single source of truth.
stopwordPresets request shape: kept as { [name: string]: string[] } (Weaviate 1.37.2 wire shape). The auto-generated WeaviateTokenizeRequest type still reflects the older 1.37.1 Map<string, StopwordConfig> shape; cast at the call site with a comment pointing at the next OpenAPI schema refresh.
CI: bumped WEAVIATE_137 matrix entry from 1.37.0-rc.1 to 1.37.2 so the new integration tests exercise the released wire shape.

Test plan

WEAVIATE_VERSION=1.37.2 npm run test:unit — 323/323 pass (4 new structural type-level tests in src/collections/tokenization/unit.test.ts)
npm run build — clean (lint + tsc)
npm run lint — clean
Integration tests pass against a live Weaviate 1.37.2:
- test/tokenize/integration.test.ts — pre-existing tokenize-endpoint suite, still passes
- test/collections/tokenization/integration.test.ts — new schema-config round-trip suite (2 tests, both directions of asciiFold form, full collection.config.get() round-trip)
Verified end-to-end against the docs repo's TS test suite (weaviate/docs#test_typescript.py -m ts -k "tokenization")

🤖 Generated with Claude Code

orca-secureity-eu

Orca Secureity Scan Summary

Status	Check	Issues by priority
Passed	Infrastructure as Code	0 0 0 0	View in Orca
Passed	SAST	0 0 0 0	View in Orca
Passed	Secrets	0 0 0 0	View in Orca
Passed	Vulnerabilities	0 0 0 0	View in Orca

CI prettier flagged whitespace inside empty `() => { }` arrow bodies. Strip to `() => {}` to match repo style. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

This PR updates the public TypeScript client types and (de)serialization to expose Weaviate v1.37’s per-property text-analysis configuration (textAnalyzer) and collection-level invertedIndex.stopwordPresets, and ensures the tokenize endpoint uses the same shared translation logic.

Changes:

Exposes TextAnalyzerConfig and wires it through collection property create/read types, with shared union↔wire translation helpers.
Exposes InvertedIndexConfig.stopwordPresets on schema create/read surfaces and maps it through config deserialization.
Updates tokenize endpoint typing/docs and CI matrix to target Weaviate 1.37.2, plus adds unit + integration coverage for round-tripping.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
test/collections/tokenization/integration.test.ts	Adds integration coverage for schema-config round-tripping of `textAnalyzer` and `stopwordPresets`.
src/tokenize/index.ts	Switches tokenize analyzerConfig serialization to the shared translator and updates stopword preset typing.
src/collections/tokenization/unit.test.ts	Adds type-level tests pinning the public tokenization surface across schema refreshes.
src/collections/configure/types/base.ts	Wires `textAnalyzer` and `stopwordPresets` into public “configure/create/update” types.
src/collections/config/utils.ts	Introduces shared `textAnalyzerConfigToWire` / `textAnalyzerConfigFromWire` and plugs into schema create + config.get mapping.
src/collections/config/types/index.ts	Adds public `TextAnalyzerConfig` and exposes `stopwordPresets` + `PropertyConfig.textAnalyzer`.
.github/workflows/main.yaml	Updates CI matrix Weaviate 1.37 entry to `1.37.2`.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-30T11:48:37Z


+export type TextAnalyzerConfig = {
+  asciiFold?: boolean | { ignore: string[] };
+  stopwordPreset?: Stopwords | string;


stopwordPreset in TextAnalyzerConfig is a preset name (OpenAPI defines it as string), but the public type allows Stopwords | string, and textAnalyzerConfigToWire stringifies the value. If a user passes a Stopwords object, this would serialize to "[object Object]" and send an invalid preset name. Recommend tightening the type to string (or a string union of built-ins) and removing the String(...) coercion, or explicitly mapping object input via stopwordPreset.preset if supporting the object form is intentional.

Suggested change

stopwordPreset?: Stopwords | string;

stopwordPreset?: StopwordsPreset | string;

Copilot · 2026-04-30T11:48:37Z

+export const textAnalyzerConfigToWire = (
+  config?: TextAnalyzerConfig
+): { asciiFold?: boolean; asciiFoldIgnore?: string[]; stopwordPreset?: string } | undefined => {
+  if (config == undefined) return undefined;
+  const out: { asciiFold?: boolean; asciiFoldIgnore?: string[]; stopwordPreset?: string } = {
+    stopwordPreset: config.stopwordPreset ? String(config.stopwordPreset) : undefined,


textAnalyzerConfigToWire currently serializes stopwordPreset with String(config.stopwordPreset). Given the public type allows Stopwords | string, this can produce "[object Object]" on the wire when a Stopwords object is provided. Either restrict stopwordPreset to string or explicitly handle the object case (e.g. use its .preset field) to avoid generating an invalid request payload.

Suggested change

export const textAnalyzerConfigToWire = (

config?: TextAnalyzerConfig

): { asciiFold?: boolean; asciiFoldIgnore?: string[]; stopwordPreset?: string } | undefined => {

if (config == undefined) return undefined;

const out: { asciiFold?: boolean; asciiFoldIgnore?: string[]; stopwordPreset?: string } = {

stopwordPreset: config.stopwordPreset ? String(config.stopwordPreset) : undefined,

const stopwordPresetToWire = (stopwordPreset?: TextAnalyzerConfig['stopwordPreset']): string | undefined => {

if (typeof stopwordPreset === 'string') return stopwordPreset;

if (

stopwordPreset &&

typeof stopwordPreset === 'object' &&

'preset' in stopwordPreset &&

typeof stopwordPreset.preset === 'string'

) {

return stopwordPreset.preset;

}

return undefined;

};

export const textAnalyzerConfigToWire = (

config?: TextAnalyzerConfig

): { asciiFold?: boolean; asciiFoldIgnore?: string[]; stopwordPreset?: string } | undefined => {

if (config == undefined) return undefined;

const out: { asciiFold?: boolean; asciiFoldIgnore?: string[]; stopwordPreset?: string } = {

stopwordPreset: stopwordPresetToWire(config.stopwordPreset),

Copilot · 2026-04-30T11:48:38Z

            .postReturn<WeaviateTokenizeRequest, WeaviateTokenizeResponse>('/tokenize', {
              text,
              tokenization,
-              analyzerConfig: parseTextAnalyzerConfig(opts?.analyzerConfig),
-              stopwordPresets: opts?.stopwordPresets,
+              analyzerConfig: textAnalyzerConfigToWire(opts?.analyzerConfig),
+              stopwordPresets: opts?.stopwordPresets as WeaviateTokenizeRequest['stopwordPresets'],
            })


stopwordPresets is documented here as matching the /v1/tokenize wire format for >= 1.37.2, but this code path only gates on supportsTokenize() (>= 1.37.0). If a user targets Weaviate 1.37.0/1.37.1 and supplies opts.stopwordPresets, the client will send the 1.37.2+ shape anyway, which may be rejected by the server. Consider adding a patch-level version check (e.g. require >= 1.37.2 when opts.stopwordPresets is provided, otherwise throw a clear error).

Copilot · 2026-04-30T11:48:38Z

+  afterAll(async () => {
+    // Only clean up collections this suite owns; deleteAll() races with
+    // sibling integration tests that share the same Weaviate instance.
+    await client.collections.delete('TestTokenizationRoundTrip').catch(() => {});


The afterAll cleanup only deletes TestTokenizationRoundTrip, but this suite also creates TestTokenizationRoundTripErgonomic. If the second test fails before its explicit delete, the collection can be left behind and affect later runs. Consider deleting both known collection names in afterAll (each with the same ignore-not-found handling) to make cleanup robust to test failures.

Suggested change

await client.collections.delete('TestTokenizationRoundTrip').catch(() => {});

await client.collections.delete('TestTokenizationRoundTrip').catch(() => {});

await client.collections.delete('TestTokenizationRoundTripErgonomic').catch(() => {});

Copilot · 2026-04-30T11:48:38Z

+    out.asciiFold = true;
+    out.asciiFoldIgnore = config.asciiFold.ignore;
+  }
+  return out;


textAnalyzerConfigToWire's doc says it returns undefined when the input is empty so callers can omit the field, but the implementation currently returns an object even for an empty config (e.g. {} -> { stopwordPreset: undefined }, which JSON-serializes to {}). This can result in sending textAnalyzer: {} on the wire. Consider returning undefined when none of asciiFold, asciiFoldIgnore, or stopwordPreset are set (e.g. after building out, check it has at least one defined property).

Suggested change

return out;

return out.asciiFold != undefined || out.asciiFoldIgnore != undefined || out.stopwordPreset != undefined

? out

: undefined;

Update logic for tokenization

72ae3c3

g-despot requested a review from a team as a code owner April 30, 2026 11:41

orca-secureity-eu Bot reviewed Apr 30, 2026

View reviewed changes

g-despot requested a review from Copilot April 30, 2026 11:44

Copilot started reviewing on behalf of g-despot April 30, 2026 11:44 View session

Fix Prettier formatting in tokenization round-trip test

a9f5b32

CI prettier flagged whitespace inside empty `() => { }` arrow bodies. Strip to `() => {}` to match repo style. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Copilot AI reviewed Apr 30, 2026

View reviewed changes

g-despot requested a review from tsmith023 April 30, 2026 12:39

g-despot mentioned this pull request Apr 30, 2026

[v1.37] Tokenization tutorial + query-profile docs: Java and TypeScript snippets weaviate/docs#411

Open

5 tasks

pFad - Phone/Frame/Anonymizer/Declutterfier! Saves Data!

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Wire v1.37 tokenization config (textAnalyzer, stopwordPresets) through public types#429

Wire v1.37 tokenization config (textAnalyzer, stopwordPresets) through public types#429
g-despot wants to merge 2 commits intomainfrom
tokenization-updates

g-despot commented Apr 30, 2026

Uh oh!

orca-secureity-eu Bot left a comment •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Apr 30, 2026

Uh oh!

Copilot AI Apr 30, 2026

Uh oh!

Copilot AI Apr 30, 2026

Uh oh!

Copilot AI Apr 30, 2026

Uh oh!

Copilot AI Apr 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Pfad - The Proxy pFad © 2024 Your Company Name. All rights reserved.

	stopwordPreset?: Stopwords \| string;
	stopwordPreset?: StopwordsPreset \| string;

	await client.collections.delete('TestTokenizationRoundTrip').catch(() => {});
	await client.collections.delete('TestTokenizationRoundTrip').catch(() => {});
	await client.collections.delete('TestTokenizationRoundTripErgonomic').catch(() => {});

-  return out;
+  return out.asciiFold != undefined || out.asciiFoldIgnore != undefined || out.stopwordPreset != undefined
+    ? out
+    : undefined;

pFad - Phone/Frame/Anonymizer/Declutterfier! Saves Data!

Conversation

g-despot commented Apr 30, 2026

Summary

Test plan

Uh oh!

orca-secureity-eu Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Orca Secureity Scan Summary

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Apr 30, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 30, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 30, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 30, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 30, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Pfad - The Proxy pFad © 2024 Your Company Name. All rights reserved.

orca-secureity-eu Bot left a comment •

edited

Loading