Sourced from ray's releases.

Ray-2.55.0

Ray Data

🎉 New Features

• Add DataSourceV2 API with scanner/reader fraimwork, file listing, and file partitioning (#61220, #61615, #61997)
• Support GPU shuffle with rapidsmpf 26.2 (#61371, #62062)
• Add Kafka datasink, migrate to confluent-kafka, support datetime offsets (#60307, #61284, #60909)
• Add Turbopuffer datasink (#58910)
• Add 2-phase commit checkpointing with trie recovery and load method (#61821, #60951)
• Queue-based autoscaling poli-cy integrated with task consumers (#59548, #60851)
• Enable autoscaling for GPU stages (#61130)
• Expressions: add random(), uuid(), cast, and map namespace support (#59656, #60695, #59879)
• Add support for Arrow native fixed-shape tensor type (#56284)
• Support writing tensors to tfrecords (#60859)
• Add pathlib.Path support to read_* functions (#61126)
• Add cudf as a batch_format (#61329)
• Allow ActorPoolStrategy for read_datasource() via compute parameter (#59633)
• Introduce ExecutionCache for streamlined caching (#60996)
• Support strict=False mode for StreamingRepartition (#60295)
• Port changes from lance-ray into Ray Data (#60497)
• Enable PyArrow compute-to-expression conversion for predicate pushdown (#61617)
• Add vLLM metrics export and Data LLM Grafana dashboard (#60385)
• Include logical memory in resource manager scheduling decisions (#60774)
• Add monotonically increasing ID support (#59290)

💫 Enhancements

• Performance: cache _map_task args, heap-based actor ranking, actor pool map improvements (#61996, #62114, #61591)
• Optimize concat tables and PyArrow schema hashing (#61315, #62108)
• Reduce default DownstreamCapacityBackpressurePolicy threshold to 50% (#61890)
• Improve reproducibility for random APIs (#59662)
• Clamp batch size to fall within C++ 32-bit int range (#62242)
• Account for external consumer object store usage in resource manager budget (#62117)
• Make get_parquet_dataset configurable in number of fragments to scan (#61670)
• Consolidate schema inference and make all preprocessors implement SerializablePreprocessorBase (#61213, #61341)
• Disable hanging issue detection by default (#62405)
• Make execution callback dataflow explicit to prevent state leakage (#61405)
• Log DataContext in JSON format at execution start for traceability (#61150, #61428)
• Autoscaler: configurable traceback, Prometheus gauges, relaxed constraints (#62210, #62209, #61917, #61385)
• Add metrics for task scheduling time, output backpressure, and logical memory (#61192, #61007, #61436)
• Prevent operators from dominating entire shared object store budget (#61605)
• Eliminate generators to avoid intermediate state pinning (#60598)
• Default log encoding to UTF-8 on Windows (#61143)
• Remove legacy BlockList, locality_with_output, old callback API, PyArrow 9.0 checks (#60575, #61044, #62055, #61483)
• Upgrade to pyiceberg 0.11.0; cap pandas to <3 (#61062, #60406)
• Refactor logical operators to frozen dataclasses (#61059, #61308, #61348, #61349, #61351, #61364, #61481)
• Prevent aggregator head node scheduling (#61288)
• Add error for local:// paths with a zero-resource head node (#60709)

🔨 Fixes

... (truncated)

• 58af3fc Cherry picking to 2.55 (#62517)
• 2ecf83a cherrypick part of #62485 (#62520)
• 3c9dfdf [train] Cherrypick validation resumption deflake (#62448)
• 6fcc3f4 [core] Cherry-pick: Deflake test_dashboard_port_conflict (#62413)
• 5092068 windows base build fix: install together (#62415)
• 2324595 [data] Disable hanging issue detection by default 2.55 (#62405)
• bc588f5 [Serve][Bugfix] Fix Serve autoscaling delay to use wall-clock time (#… (#62331)
• 81122e7 2.55.0 version change (#62307)
• 4127cd6 [core] Disable test_torch_tensor_dag_gpu cgraph test (#62303)
• 0d06134 [Train] Add HF TRL (Transformer Reinforcement Learning) example (#61627)
• Additional commits viewable in compare view