I will only approve this PR if you commit to keep this file up to date when the rest of the code changes.

A single big CLAUDE.md file like this does not seem useful, in fact it looks like it's actively harmful the tool even displays a warning about the file being too large. It is also unlikely that most people working with claude will use your provided scripts to split the files

I tried to approach of splitting it up too and I did not notice any difference in quality of content it produces. FWIW it seems to look at README.mds too so maybe it's more helpful to write good and useful README files and tell claude to look at @docs.feldera.com, @openapi.json and README files.

I suggest we just put all higher level CLAUDE.md files in gitignore and everyone can have whatever CLAUDE.md files in higher level directories they want. You can also put it as a commit that can be applied to any branch someone works in if you like but I really don't see value in having this comitted to main. I think this is snake oil at this point without any objective measure whether it helps with anything or not.

I have been using coding agents when working on NATS connector and exploring Feldera code base. When pulling a fresh Feldera main I delete the CLAUDE.md file from Feldera repo. Is is too big and and make prompts expensive and agent get unfocused with to much info. But that just my experience.

Anthropic recommend it to be short and focus. https://code.claude.com/docs/en/best-practices#write-an-effective-claude-md

But you can also make CLAUDE.md files in subfolders that the agent will read if it visits the folder.

And for some tasks and domain knowledge. Skills can be interesting. I playing with a NATS output connector made with a output connector skill that the agent made before implementing the connector.