[Git][debian-gis-team/fiona][experimental] 10 commits: Add upstream patch to fix FTBFS with GCC 14. (closes: #1074959)
Bas Couwenberg (@sebastic)
gitlab at salsa.debian.org
Tue Jul 30 16:44:47 BST 2024
Bas Couwenberg pushed to branch experimental at Debian GIS Project / fiona
Commits:
548bf297 by Bas Couwenberg at 2024-07-28T19:15:00+02:00
Add upstream patch to fix FTBFS with GCC 14. (closes: #1074959)
- - - - -
4a198a8c by Bas Couwenberg at 2024-07-28T19:23:21+02:00
Update lintian overrides.
- - - - -
78593769 by Bas Couwenberg at 2024-07-28T19:24:57+02:00
Bump Standards-Version to 4.7.0, no changes.
- - - - -
32be2745 by Bas Couwenberg at 2024-07-28T19:24:57+02:00
Set distribution to unstable.
- - - - -
15eaed51 by Bas Couwenberg at 2024-07-28T19:31:31+02:00
Merge tag 'debian/1.9.6-2' into experimental
releasing package fiona version 1.9.6-2
- - - - -
95eef959 by Bas Couwenberg at 2024-07-30T17:33:16+02:00
New upstream version 1.10~b3
- - - - -
5115d034 by Bas Couwenberg at 2024-07-30T17:33:18+02:00
Update upstream source from tag 'upstream/1.10_b3'
Update to upstream version '1.10~b3'
with Debian dir 3628c363323766965a799b7751f74e130ac230ca
- - - - -
b8524a2b by Bas Couwenberg at 2024-07-30T17:34:20+02:00
New upstream beta release.
- - - - -
a2804ac3 by Bas Couwenberg at 2024-07-30T17:35:33+02:00
Drop test_opener_fsspec_file_fs_listdir.patch, applied upstream.
- - - - -
02911509 by Bas Couwenberg at 2024-07-30T17:35:52+02:00
Set distribution to experimental.
- - - - -
12 changed files:
- .github/workflows/scorecard.yml
- CHANGES.txt
- Makefile
- debian/changelog
- debian/control
- debian/patches/series
- − debian/patches/test_opener_fsspec_file_fs_listdir.patch
- + debian/source/lintian-overrides
- fiona/__init__.py
- fiona/_vsiopener.pyx
- + fiona/abc.py
- tests/test_pyopener.py
Changes:
=====================================
.github/workflows/scorecard.yml
=====================================
@@ -37,7 +37,7 @@ jobs:
persist-credentials: false
- name: "Run analysis"
- uses: ossf/scorecard-action at dc50aa9510b46c811795eb24b2f1ba02a914e534 # v2.3.3
+ uses: ossf/scorecard-action at 62b2cac7ed8198b15735ed49ab1e5cf35480ba46 # v2.4.0
with:
results_file: results.sarif
results_format: sarif
@@ -67,6 +67,6 @@ jobs:
# Upload the results to GitHub's code scanning dashboard.
- name: "Upload to code-scanning"
- uses: github/codeql-action/upload-sarif at b611370bb5703a7efb587f9d136a52ea24c5c38c # v3.25.11
+ uses: github/codeql-action/upload-sarif at afb54ba388a7dca6ecae48f608c4ff05ff4cc77a # v3.25.15
with:
sarif_file: results.sarif
=====================================
CHANGES.txt
=====================================
@@ -3,6 +3,15 @@ Changes
All issue numbers are relative to https://github.com/Toblerity/Fiona/issues.
+1.10b3 (2024-07-29)
+-------------------
+
+Bug fixes:
+
+- The sketchy, semi-private Python opener interfaces of version 1.10b2 have
+ been replaced by ABCs that are exported from fiona.abc (#1415).
+- The truncate VSI plugin callback has been implemented (#1413).
+
1.10b2 (2024-07-10)
-------------------
=====================================
Makefile
=====================================
@@ -1,5 +1,5 @@
PYTHON_VERSION ?= 3.10
-GDAL ?= ubuntu-small-3.6.4
+GDAL ?= ubuntu-small-3.9.0
all: deps clean install test
.PHONY: docs
@@ -51,4 +51,4 @@ dockertestimage-amd64:
docker build --platform linux/amd64 --target gdal --build-arg GDAL=$(GDAL) --build-arg PYTHON_VERSION=$(PYTHON_VERSION) -t fiona-amd64:$(GDAL)-py$(PYTHON_VERSION) .
dockertest-amd64: dockertestimage-amd64
- docker run -it -v $(shell pwd):/app -v /tmp:/tmp --env AWS_ACCESS_KEY_ID --env AWS_SECRET_ACCESS_KEY --entrypoint=/bin/bash fiona-amd64:$(GDAL)-py$(PYTHON_VERSION) -c '/venv/bin/python -m pip install --editable .[all] --no-build-isolation && /venv/bin/python -B -m pytest -m "not wheel" --cov fiona --cov-report term-missing $(OPTS)'
+ docker run -it -v $(shell pwd):/app -v /tmp:/tmp --env AWS_ACCESS_KEY_ID --env AWS_SECRET_ACCESS_KEY --entrypoint=/bin/bash fiona-amd64:$(GDAL)-py$(PYTHON_VERSION) -c '/venv/bin/python -m pip install tiledb && /venv/bin/python -m pip install --editable .[all] --no-build-isolation && /venv/bin/python -B -m pytest -m "not wheel" --cov fiona --cov-report term-missing $(OPTS)'
=====================================
debian/changelog
=====================================
@@ -1,3 +1,11 @@
+fiona (1.10~b3-1~exp1) experimental; urgency=medium
+
+ * Team upload.
+ * New upstream beta release.
+ * Drop test_opener_fsspec_file_fs_listdir.patch, applied upstream.
+
+ -- Bas Couwenberg <sebastic at debian.org> Tue, 30 Jul 2024 17:35:38 +0200
+
fiona (1.10~b2-1~exp1) experimental; urgency=medium
* Team upload.
@@ -24,6 +32,16 @@ fiona (1.10~b1-1~exp1) experimental; urgency=medium
-- Bas Couwenberg <sebastic at debian.org> Wed, 17 Apr 2024 06:14:12 +0200
+fiona (1.9.6-2) unstable; urgency=medium
+
+ * Team upload.
+ * Add upstream patch to fix FTBFS with GCC 14.
+ (closes: #1074959)
+ * Update lintian overrides.
+ * Bump Standards-Version to 4.7.0, no changes.
+
+ -- Bas Couwenberg <sebastic at debian.org> Sun, 28 Jul 2024 19:14:33 +0200
+
fiona (1.9.6-1) unstable; urgency=medium
* Team upload.
=====================================
debian/control
=====================================
@@ -26,7 +26,7 @@ Build-Depends: debhelper-compat (= 13),
python3-setuptools,
python3-shapely,
python3-tz <!nocheck>,
-Standards-Version: 4.6.2
+Standards-Version: 4.7.0
Vcs-Browser: https://salsa.debian.org/debian-gis-team/fiona
Vcs-Git: https://salsa.debian.org/debian-gis-team/fiona.git -b experimental
Homepage: https://github.com/Toblerity/Fiona
=====================================
debian/patches/series
=====================================
@@ -1,3 +1,2 @@
0001-Rename-fio-command-to-fiona-to-avoid-name-clash.patch
test_drvsupport.patch
-test_opener_fsspec_file_fs_listdir.patch
=====================================
debian/patches/test_opener_fsspec_file_fs_listdir.patch deleted
=====================================
@@ -1,16 +0,0 @@
-Description: Fix test failure.
-Author: Bas Couwenberg <sebastic at debian.org>
-Bug: https://github.com/Toblerity/Fiona/issues/1332
-Applied-Upstream: https://github.com/Toblerity/Fiona/commit/afdd0e79cc2d34f90727bf3caaf229fe8d220950
-
---- a/tests/test_pyopener.py
-+++ b/tests/test_pyopener.py
-@@ -143,7 +143,7 @@ def test_opener_fsspec_file_fs_listdir()
- """Use fsspec file filesystem as opener for listdir()."""
- fs = fsspec.filesystem("file")
- listing = fiona.listdir("tests/data", opener=fs)
-- assert len(listing) >= 35
-+ assert len(listing) >= 33
- assert set(
- ["coutwildrnp.shp", "coutwildrnp.dbf", "coutwildrnp.shx", "coutwildrnp.prj"]
- ) & set(listing)
=====================================
debian/source/lintian-overrides
=====================================
@@ -0,0 +1,3 @@
+# False positive, uuid is not deprecated
+uses-deprecated-python-stdlib uu *
+
=====================================
fiona/__init__.py
=====================================
@@ -78,7 +78,7 @@ __all__ = [
"remove",
]
-__version__ = "1.10b2"
+__version__ = "1.10b3"
__gdal_version__ = get_gdal_release_name()
gdal_version = get_gdal_version_tuple()
=====================================
fiona/_vsiopener.pyx
=====================================
@@ -3,8 +3,11 @@
Based on _filepath.pyx.
"""
+from abc import ABC, abstractmethod
+from collections.abc import Callable
import contextlib
from contextvars import ContextVar
+from functools import singledispatch
import logging
import os
from pathlib import Path
@@ -288,7 +291,8 @@ cdef size_t pyopener_write(void *pFile, void *pBuffer, size_t nSize, size_t nCou
"Writing data: file_obj=%r, buff_view=%r, buffer_len=%r",
file_obj,
buff_view,
- buffer_len)
+ buffer_len
+ )
try:
num = file_obj.write(buff_view)
except TypeError:
@@ -306,6 +310,16 @@ cdef int pyopener_flush(void *pFile) with gil:
return 1
+cdef int pyopener_truncate(void *pFile, vsi_l_offset size) with gil:
+ cdef object file_obj = <object>pFile
+ log.debug("Truncating: file_obj=%r, size=%r", file_obj, size)
+ try:
+ file_obj.truncate(size)
+ return 0
+ except AttributeError:
+ return 1
+
+
cdef int pyopener_close(void *pFile) with gil:
cdef object file_obj = <object>pFile
log.debug("Closing: file_obj=%r", file_obj)
@@ -316,6 +330,26 @@ cdef int pyopener_close(void *pFile) with gil:
return 0
+cdef int pyopener_read_multi_range(void *pFile, int nRanges, void **ppData, vsi_l_offset *panOffsets, size_t *panSizes) except -1 with gil:
+ cdef object file_obj = <object>pFile
+
+ if not hasattr(file_obj, "read_multi_range"):
+ errmsg = "MultiRangeRead not implemented for Opener".encode("utf-8")
+ CPLError(CE_Failure, <CPLErrorNum>1, <const char *>"%s", <const char *>errmsg)
+ return -1
+
+ # NOTE: Convert panOffsets and panSizes to Python lists
+ cdef list offsets = [int(panOffsets[i]) for i in range(nRanges)]
+ cdef list sizes = [int(panSizes[i]) for i in range(nRanges)]
+
+ # NOTE: Call the Python method with the converted arguments
+ cdef list python_data = file_obj.read_multi_range(nRanges, offsets, sizes)
+ for i in range(nRanges):
+ memcpy(ppData[i], <void*><char*>python_data[i], len(python_data[i]))
+
+ return 0
+
+
@contextlib.contextmanager
def _opener_registration(urlpath, obj):
cdef char **registered_prefixes = NULL
@@ -342,7 +376,17 @@ def _opener_registration(urlpath, obj):
cdef bytes prefix_bytes = f"/{namespace}/".encode("utf-8")
# Might raise.
- opener = _create_opener(obj)
+ opener = to_pyopener(obj)
+
+ # Before returning we do a quick check that the opener will
+ # plausibly function.
+ try:
+ _ = opener.size("test")
+ except (AttributeError, TypeError, ValueError) as err:
+ raise OpenerRegistrationError(f"Opener is invalid.") from err
+ except Exception:
+ # We expect the path to not resolve.
+ pass
registry = _OPENER_REGISTRY.get({})
@@ -373,11 +417,17 @@ def _opener_registration(urlpath, obj):
callbacks_struct.read = <VSIFilesystemPluginReadCallback>pyopener_read
callbacks_struct.write = <VSIFilesystemPluginWriteCallback>pyopener_write
callbacks_struct.flush = <VSIFilesystemPluginFlushCallback>pyopener_flush
+ callbacks_struct.truncate = <VSIFilesystemPluginTruncateCallback>pyopener_truncate
callbacks_struct.close = <VSIFilesystemPluginCloseCallback>pyopener_close
callbacks_struct.read_dir = <VSIFilesystemPluginReadDirCallback>pyopener_read_dir
callbacks_struct.stat = <VSIFilesystemPluginStatCallback>pyopener_stat
callbacks_struct.unlink = <VSIFilesystemPluginUnlinkCallback>pyopener_unlink
+
+ if isinstance(opener, MultiByteRangeResource):
+ callbacks_struct.read_multi_range = <VSIFilesystemPluginReadMultiRangeCallback>pyopener_read_multi_range
+
callbacks_struct.pUserData = &fsdata
+
retval = VSIInstallPluginHandler(prefix_bytes, callbacks_struct)
VSIFreeFilesystemPluginCallbacksStruct(callbacks_struct)
@@ -399,9 +449,10 @@ def _opener_registration(urlpath, obj):
retval = VSIRemovePluginHandler(prefix_bytes)
-class _AbstractOpener:
- """Adapts a Python object to the opener interface."""
- def open(self, path, mode="r", **kwds):
+class FileContainer(ABC):
+ """An object that can report on and open Python files."""
+ @abstractmethod
+ def open(self, path: str, mode: str = "r", **kwds):
"""Get a Python file object for a resource.
Parameters
@@ -419,8 +470,10 @@ class _AbstractOpener:
A Python 'file' object with methods read/write, seek, tell,
etc.
"""
- raise NotImplementedError
- def isfile(self, path):
+ pass
+
+ @abstractmethod
+ def isfile(self, path: str) -> bool:
"""Test if the resource is a 'file', a sequence of bytes.
Parameters
@@ -432,8 +485,10 @@ class _AbstractOpener:
-------
bool
"""
- raise NotImplementedError
- def isdir(self, path):
+ pass
+
+ @abstractmethod
+ def isdir(self, path: str) -> bool:
"""Test if the resource is a 'directory', a container.
Parameters
@@ -445,8 +500,10 @@ class _AbstractOpener:
-------
bool
"""
- raise NotImplementedError
- def ls(self, path):
+ pass
+
+ @abstractmethod
+ def ls(self, path: str) -> list[str]:
"""Get a 'directory' listing.
Parameters
@@ -459,8 +516,10 @@ class _AbstractOpener:
list of str
List of 'path' paths relative to the directory.
"""
- raise NotImplementedError
- def mtime(self, path):
+ pass
+
+ @abstractmethod
+ def mtime(self, path: str) -> int:
"""Get the mtime of a resource..
Parameters
@@ -473,8 +532,10 @@ class _AbstractOpener:
int
Modification timestamp in seconds.
"""
- raise NotImplementedError
- def rm(self, path):
+ pass
+
+ @abstractmethod
+ def rm(self, path: str) -> None:
"""Remove a resource.
Parameters
@@ -486,8 +547,10 @@ class _AbstractOpener:
-------
None
"""
- raise NotImplementedError
- def size(self, path):
+ pass
+
+ @abstractmethod
+ def size(self, path: str) -> int:
"""Get the size, in bytes, of a resource..
Parameters
@@ -499,10 +562,26 @@ class _AbstractOpener:
-------
int
"""
- raise NotImplementedError
+ pass
+
+
+class MultiByteRangeResource(ABC):
+ """An object that provides VSIFilesystemPluginReadMultiRangeCallback."""
+ @abstractmethod
+ def get_byte_ranges(self, offsets: list[int], sizes: list[int]) -> list[bytes]:
+ """Get a sequence of bytes specified by a sequence of ranges."""
+ pass
+
+class MultiByteRangeResourceContainer(FileContainer):
+ """An object that can open a MultiByteRangeResource."""
+ @abstractmethod
+ def open(self, path: str, **kwds) -> MultiByteRangeResource:
+ """Open the resource at the given path."""
+ pass
-class _FileOpener(_AbstractOpener):
+
+class _FileContainer(FileContainer):
"""Adapts a Python file object to the opener interface."""
def __init__(self, obj):
self._obj = obj
@@ -516,13 +595,15 @@ class _FileOpener(_AbstractOpener):
return []
def mtime(self, path):
return 0
+ def rm(self, path):
+ pass
def size(self, path):
with self._obj(path) as f:
f.seek(0, os.SEEK_END)
return f.tell()
-class _FilesystemOpener(_AbstractOpener):
+class _FilesystemContainer(FileContainer):
"""Adapts an fsspec filesystem object to the opener interface."""
def __init__(self, obj):
self._obj = obj
@@ -547,7 +628,7 @@ class _FilesystemOpener(_AbstractOpener):
return self._obj.size(path)
-class _AltFilesystemOpener(_FilesystemOpener):
+class _AltFilesystemContainer(_FilesystemContainer):
"""Adapts a tiledb virtual filesystem object to the opener interface."""
def isfile(self, path):
return self._obj.is_file(path)
@@ -561,25 +642,20 @@ class _AltFilesystemOpener(_FilesystemOpener):
return self._obj.file_size(path)
-def _create_opener(obj):
- """Adapt Python file and fsspec objects to the opener interface."""
- if isinstance(obj, _AbstractOpener):
- opener = obj
- elif callable(obj):
- opener = _FileOpener(obj)
- elif hasattr(obj, "file_size"):
- opener = _AltFilesystemOpener(obj)
+ at singledispatch
+def to_pyopener(obj):
+ """Adapt an object to the Pyopener interface."""
+ if hasattr(obj, "file_size"):
+ return _AltFilesystemContainer(obj)
else:
- opener = _FilesystemOpener(obj)
+ return _FilesystemContainer(obj)
+
+
+ at to_pyopener.register(FileContainer)
+def _(obj):
+ return obj
- # Before returning we do a quick check that the opener will
- # plausibly function.
- try:
- _ = opener.size("test")
- except (AttributeError, TypeError, ValueError) as err:
- raise OpenerRegistrationError(f"Opener is invalid.") from err
- except Exception:
- # We expect the path to not resolve.
- pass
- return opener
+ at to_pyopener.register(Callable)
+def _(obj):
+ return _FileContainer(obj)
=====================================
fiona/abc.py
=====================================
@@ -0,0 +1,3 @@
+"""Abstract base classes."""
+
+from fiona._vsiopener import FileContainer, MultiByteRangeResourceContainer
=====================================
tests/test_pyopener.py
=====================================
@@ -138,12 +138,11 @@ def test_opener_fsspec_zip_fs_listdir():
) & set(listing)
-
def test_opener_fsspec_file_fs_listdir():
"""Use fsspec file filesystem as opener for listdir()."""
fs = fsspec.filesystem("file")
listing = fiona.listdir("tests/data", opener=fs)
- assert len(listing) >= 35
+ assert len(listing) >= 33
assert set(
["coutwildrnp.shp", "coutwildrnp.dbf", "coutwildrnp.shx", "coutwildrnp.prj"]
) & set(listing)
@@ -161,3 +160,45 @@ def test_opener_fsspec_file_remove(data):
assert not set(
["coutwildrnp.shp", "coutwildrnp.dbf", "coutwildrnp.shx", "coutwildrnp.prj"]
) & set(listing)
+
+
+def test_opener_tiledb_vfs_listdir():
+ """Use tiledb VFS as opener for listdir()."""
+ tiledb = pytest.importorskip("tiledb")
+ fs = tiledb.VFS()
+ listing = fiona.listdir("tests/data", opener=fs)
+ assert len(listing) >= 33
+ assert set(
+ ["coutwildrnp.shp", "coutwildrnp.dbf", "coutwildrnp.shx", "coutwildrnp.prj"]
+ ) & set(listing)
+
+
+def test_opener_interface():
+ """Demonstrate implementation of a custom opener."""
+ import pathlib
+ from fiona.abc import FileContainer
+
+ class CustomContainer:
+ """GDAL's VSI ReadDir() uses 5 of FileContainer's methods."""
+ def isdir(self, path):
+ return pathlib.Path(path).is_dir()
+
+ def isfile(self, path):
+ return pathlib.Path(path).is_file()
+
+ def ls(self, path):
+ return list(pathlib.Path(path).iterdir())
+
+ def mtime(self, path):
+ return pathlib.Path(path).stat().st_mtime
+
+ def size(self, path):
+ return pathlib.Path(path).stat().st_size
+
+ FileContainer.register(CustomContainer)
+
+ listing = fiona.listdir("tests/data", opener=CustomContainer())
+ assert len(listing) >= 33
+ assert set(
+ ["coutwildrnp.shp", "coutwildrnp.dbf", "coutwildrnp.shx", "coutwildrnp.prj"]
+ ) & set(listing)
View it on GitLab: https://salsa.debian.org/debian-gis-team/fiona/-/compare/6f1f3e58821fe91ee3ffc69b103b76a3f50be884...02911509a935d752b4fcd4684caa7f4c82b44b25
--
View it on GitLab: https://salsa.debian.org/debian-gis-team/fiona/-/compare/6f1f3e58821fe91ee3ffc69b103b76a3f50be884...02911509a935d752b4fcd4684caa7f4c82b44b25
You're receiving this email because of your account on salsa.debian.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/pkg-grass-devel/attachments/20240730/c39db191/attachment-0001.htm>
More information about the Pkg-grass-devel
mailing list