-
|
Is it possible to ingest data into Feldera via HTTP API in a way that is reliable to network failures? E.g. if my HTTP request fails, how do I know whether it failed before the data was ingested (and I should retry) or after the data was ingested (and I shouldn't retry)? |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments
-
|
If HTTP is the only data source feeding the pipeline, you can achieve this by tracking the number of records ingested by the pipeline via the |
Beta Was this translation helpful? Give feedback.
-
|
@zielmicha if your table has primary keys and you retry the HTTP request it should be idempotent (set semantics). |
Beta Was this translation helpful? Give feedback.
If HTTP is the only data source feeding the pipeline, you can achieve this by tracking the number of records ingested by the pipeline via the
/v0/pipelines/<pipeline_name>/statsendpoint. Theglobal_metrics.total_input_recordscounter tracks the total number of records ingested by the pipeline. If the HTTP connector has successfully ingested the data, this counter will increment by the number of records in the HTTP request.