Add HTTP body compression between pipeline-manager and pipelines, raise the limit#5626
Add HTTP body compression between pipeline-manager and pipelines, raise the limit#5626Karakatiza666 wants to merge 3 commits intomainfrom
Conversation
Karakatiza666
force-pushed
the
issue5624
branch
from
0356b4e to
77ce2ee
Compare
|
This needs unit and integration tests. I'm also nervous about disabling compression on proxied streaming requests. |
|
The compression was already disabled for streaming requests because it breaks the on-demand data streaming - the stream does not send the data as soon as it's available and instead buffers to collect a complete encoding chunk |
| .headers() | ||
| .into_iter() | ||
| .filter(|(h, _)| *h != "connection") | ||
| .filter(|(h, _)| *h != "connection" && *h != "accept-encoding") |
There was a problem hiding this comment.
This looks too subtle. I hope we find a more bullet-proof way not to compress stuff that shouldn't be compressed.
There was a problem hiding this comment.
Well, for now we have abstracted the proxying to the pipeline process behind two functions - streaming and non-streaming
…r and pipelines, raise the limit Signed-off-by: Karakatiza666 <bulakh.96@gmail.com>
…ests Signed-off-by: Karakatiza666 <bulakh.96@gmail.com>
Karakatiza666
force-pushed
the
issue5624
branch
from
77ce2ee to
2c83558
Compare
|
Based on the feedback in #5624: Added unit and integration tests that confirm compression and streaming behavior of circuit_profile endpoints |
Signed-off-by: feldera-bot <feldera-bot@feldera.com>
4 participants
The issue was caused by the artifical RESPONSE_SIZE_LIMIT for the uncompressed response body size for requests from pipeline-manager to the pipeline server.
Fix #5624: [PROFILE] The circuit profile should be compressed
Testing:
With an artificially lowered RESPONSE_SIZE_LIMIT I saw the same error when downloading the support bundle for a small pipeline.
With these changes, after recompiling the pipeline-manager and the pipeline Performance, Logs and Changes Stream tabs that use proxied streaming requests worked as expected, and I was able to download the support bundle with valid contents.