[med-svn] [Git][med-team/libatomic-queue][master] 13 commits: Versioned Build-Depends: libxenium-dev (>= 0.0.2+ds-2~)
Andreas Tille
gitlab at salsa.debian.org
Sat Dec 5 20:09:20 GMT 2020
Andreas Tille pushed to branch master at Debian Med / libatomic-queue
Commits:
688d7366 by Andreas Tille at 2020-12-05T20:38:44+01:00
Versioned Build-Depends: libxenium-dev (>= 0.0.2+ds-2~)
- - - - -
5d036ed5 by Andreas Tille at 2020-12-05T20:39:41+01:00
routine-update: New upstream version
- - - - -
2b1e15cf by Andreas Tille at 2020-12-05T20:39:42+01:00
New upstream version 0.0+git20201108.d9d66b6
- - - - -
79d7b29a by Andreas Tille at 2020-12-05T20:40:16+01:00
Update upstream source from tag 'upstream/0.0+git20201108.d9d66b6'
Update to upstream version '0.0+git20201108.d9d66b6'
with Debian dir 3852176a97489a7e83f10cc35e6a210d16af6faa
- - - - -
08b246cd by Andreas Tille at 2020-12-05T20:40:16+01:00
routine-update: Standards-Version: 4.5.1
- - - - -
e8836456 by Andreas Tille at 2020-12-05T20:40:16+01:00
routine-update: debhelper-compat 13
- - - - -
a0446aed by Andreas Tille at 2020-12-05T20:40:23+01:00
routine-update: Remove trailing whitespace in debian/rules
- - - - -
6a474591 by Andreas Tille at 2020-12-05T20:40:24+01:00
Trim trailing whitespace.
Changes-By: lintian-brush
Fixes: lintian: trailing-whitespace
See-also: https://lintian.debian.org/tags/trailing-whitespace.html
- - - - -
1c427bda by Andreas Tille at 2020-12-05T20:40:26+01:00
Set upstream metadata fields: Bug-Database, Bug-Submit.
Changes-By: lintian-brush
Fixes: lintian: upstream-metadata-file-is-missing
See-also: https://lintian.debian.org/tags/upstream-metadata-file-is-missing.html
Fixes: lintian: upstream-metadata-missing-bug-tracking
See-also: https://lintian.debian.org/tags/upstream-metadata-missing-bug-tracking.html
- - - - -
8904bc1e by Andreas Tille at 2020-12-05T20:40:27+01:00
Remove Section on libatomic-queue0 that duplicates source.
Changes-By: lintian-brush
Fixes: lintian: binary-control-field-duplicates-source
See-also: https://lintian.debian.org/tags/binary-control-field-duplicates-source.html
- - - - -
e0d09182 by Andreas Tille at 2020-12-05T20:50:32+01:00
Adapt patches
- - - - -
ab5810a6 by Andreas Tille at 2020-12-05T20:59:45+01:00
Re-add "Section: libs"
- - - - -
60514da9 by Andreas Tille at 2020-12-05T21:01:15+01:00
Upload to unstable
- - - - -
10 changed files:
- .github/workflows/c-cpp.yml
- Makefile
- README.md
- debian/changelog
- debian/control
- debian/patches/generate-shared-library.patch
- debian/rules
- + debian/upstream/metadata
- include/atomic_queue/atomic_queue.h
- src/example.cc
Changes:
=====================================
.github/workflows/c-cpp.yml
=====================================
@@ -16,6 +16,6 @@ jobs:
- name: Environment variables
run: make env; make TOOLSET=gcc versions; make TOOLSET=clang versions
- name: Unit tests with gcc
- run: make -rj2 TOOLSET=gcc run_tests
+ run: make -rj2 TOOLSET=gcc example run_tests
- name: Unit tests with clang
- run: make -rj2 TOOLSET=clang run_tests
+ run: make -rj2 TOOLSET=clang example run_tests
=====================================
Makefile
=====================================
@@ -69,7 +69,7 @@ LINK.EXE = ${LD} -o $@ $(ldflags) $(filter-out Makefile,$^) $(ldlibs)
LINK.SO = ${LD} -o $@ -shared $(ldflags) $(filter-out Makefile,$^) $(ldlibs)
LINK.A = ${AR} rscT $@ $(filter-out Makefile,$^)
-exes := benchmarks tests
+exes := benchmarks tests example
all : ${exes}
=====================================
README.md
=====================================
@@ -3,16 +3,14 @@
# atomic_queue
C++14 multiple-producer-multiple-consumer *lockless* queues based on circular buffer with [`std::atomic`][3].
-The main design principle these queues follow is _simplicity_: the bare minimum of atomic operations, fixed size buffer, value semantics.
-
-The circular buffer side-steps the memory reclamation problem inherent in linked-list based queues for the price of fixed buffer size. See [Effective memory reclamation for lock-free data structures in C++][4] for more details.
+The main design principle these queues follow is _minimalism_: the bare minimum of atomic operations, fixed size buffer, value semantics.
These qualities are also limitations:
-* The maximum queue size must be set at compile time or construction time.
-* There are no OS-blocking push/pop functions.
+* The maximum queue size must be set at compile time or construction time. The circular buffer side-steps the memory reclamation problem inherent in linked-list based queues for the price of fixed buffer size. See [Effective memory reclamation for lock-free data structures in C++][4] for more details. Fixed buffer size may not be that much of a limitation, since once the queue gets larger than the maximum expected size that indicates a problem that elements aren't processed fast enough, and if the queue keeps growing it may eventually consume all available memory which may affect the entire system, rather than the problematic process only. The only apparent inconvenience is that one has to do an upfront back-of-the-envelope calculation on what would be the largest expected/acceptable queue size.
+* There are no OS-blocking push/pop functions. This queue is designed for ultra-low-latency scenarios and using an OS blocking primitive would be sacrificing push-to-pop latency. For lowest possible latency one cannot afford blocking in the OS kernel because the wake-up latency of a blocked thread is about 1-3 microseconds, whereas this queue's round-trip time can be as low as 150 nanoseconds.
-Nevertheless, ultra-low-latency applications need just that and nothing more. The simplicity pays off, see the [throughput and latency benchmarks][1].
+Ultra-low-latency applications need just that and nothing more. The minimalism pays off, see the [throughput and latency benchmarks][1].
Available containers are:
* `AtomicQueue` - a fixed size ring-buffer for atomic elements.
@@ -92,7 +90,7 @@ In a production multiple-producer-multiple-consumer scenario the ring-buffer siz
Using a power-of-2 ring-buffer array size allows a couple of important optimizations:
-* The writer and reader indexes get mapped into the ring-buffer array index using modulo `% SIZE` binary operator and using a power-of-2 size turns that modulo operator into one plain `and` instruction and that is as fast as it gets.
+* The writer and reader indexes get mapped into the ring-buffer array index using remainder binary operator `% SIZE` and using a power-of-2 size turns that remainder operator into one plain `and` instruction and that is as fast as it gets.
* The *element index within the cache line* gets swapped with the *cache line index* within the *ring-buffer array element index*, so that subsequent queue elements actually reside in different cache lines. This eliminates contention between producers and consumers on the ring-buffer cache lines. Instead of `N` producers together with `M` consumers competing on the same ring-buffer array cache line in the worst case, it is only one producer competing with one consumer. This optimisation scales better with the number of producers and consumers, and element size. With low number of producers and consumers (up to about 2 of each in these benchmarks) disabling this optimisation may yield better throughput (but higher variance across runs).
The containers use `unsigned` type for size and internal indexes. On x86-64 platform `unsigned` is 32-bit wide, whereas `size_t` is 64-bit wide. 64-bit instructions utilise an extra byte instruction prefix resulting in slightly more pressure on the CPU instruction cache and the front-end. Hence, 32-bit `unsigned` indexes are used to maximise performance. That limits the queue size to 4,294,967,295 elements, which seems to be a reasonable hard limit for many applications.
@@ -141,7 +139,7 @@ The project uses `.editorconfig` and `.clang-format` to automate formatting. Pul
## Help needed
* Submit pull requests with benchmarking code for other queues. The queues should be somewhat widely used or have exceptional performance, not my-first-mpmc-queue type projects.
-* Benchmarking results on different architectures or with much more cores. Run `scripts/run-benchmarks.sh` and email me the results file, or put it under `results/` and submit a pull request.
+* Benchmarking results on different architectures or with much more cores, in particular on AMD Ryzen CPUs. Run `scripts/run-benchmarks.sh` and email me the results file, or put it under `results/` and submit a pull request.
---
=====================================
debian/changelog
=====================================
@@ -1,3 +1,18 @@
+libatomic-queue (0.0+git20201108.d9d66b6-1) unstable; urgency=medium
+
+ * New upstream commit
+ * Versioned Build-Depends: libxenium-dev (>= 0.0.2+ds-2~)
+ Closes: #976481
+ * New upstream version
+ * Standards-Version: 4.5.1 (routine-update)
+ * debhelper-compat 13 (routine-update)
+ * Remove trailing whitespace in debian/rules (routine-update)
+ * Trim trailing whitespace.
+ * Set upstream metadata fields: Bug-Database, Bug-Submit.
+ * Remove Section on libatomic-queue0 that duplicates source.
+
+ -- Andreas Tille <tille at debian.org> Sat, 05 Dec 2020 20:51:05 +0100
+
libatomic-queue (0.0+git20201007.df79403-2) unstable; urgency=medium
* Source-only-upload.
=====================================
debian/control
=====================================
@@ -4,15 +4,15 @@ Maintainer: Debian Med Packaging Team <debian-med-packaging at lists.alioth.debian.
Uploaders: Steffen Moeller <moeller at debian.org>,
Andreas Tille <tille at debian.org>,
Nilesh Patra <npatra974 at gmail.com>
-Build-Depends: debhelper-compat (= 12),
+Build-Depends: debhelper-compat (= 13),
d-shlibs,
libboost-dev,
libboost-test-dev,
libconcurrentqueue-dev,
libreaderwriterqueue-dev,
libtbb-dev,
- libxenium-dev
-Standards-Version: 4.5.0
+ libxenium-dev (>= 0.0.2+ds-2~)
+Standards-Version: 4.5.1
Section: libs
Homepage: https://github.com/max0x7ba/atomic_queue
Vcs-Browser: https://salsa.debian.org/med-team/libatomic-queue
@@ -97,4 +97,3 @@ Description: devel files for C++ atomic_queue library
specified as an argument to the constructor.
.
This package contains the header files and static library.
-
=====================================
debian/patches/generate-shared-library.patch
=====================================
@@ -5,7 +5,7 @@ Description: Fix unused variable
--- a/Makefile
+++ b/Makefile
-@@ -10,6 +10,7 @@
+@@ -10,6 +10,7 @@ BUILD := release
TOOLSET := gcc
build_dir := ${CURDIR}/build/${BUILD}/${TOOLSET}
@@ -13,7 +13,7 @@ Description: Fix unused variable
cxx.gcc := g++
cc.gcc := gcc
-@@ -61,21 +62,30 @@
+@@ -61,21 +62,30 @@ ldlibs.moodycamel :=
cppflags.xenium := -I${abspath ../xenium}
ldlibs.xenium :=
@@ -27,7 +27,7 @@ Description: Fix unused variable
+LINK.SO = ${LD} -o $@.$(SOVERSION) -shared -Wl,-soname,`basename $@`.$(SOVERSION) $(ldflags) $(filter-out Makefile,$^) $(ldlibs)
LINK.A = ${AR} rscT $@ $(filter-out Makefile,$^)
- exes := benchmarks tests
+ exes := benchmarks tests example
all : ${exes}
@@ -46,7 +46,7 @@ Description: Fix unused variable
benchmarks_src := benchmarks.cc cpu_base_frequency.cc huge_pages.cc
${build_dir}/benchmarks : cppflags += ${cppflags.tbb} ${cppflags.moodycamel} ${cppflags.xenium}
${build_dir}/benchmarks : ldlibs += ${ldlibs.tbb} ${ldlibs.moodycamel} ${ldlibs.xenium} -ldl
-@@ -95,9 +105,10 @@
+@@ -95,9 +105,10 @@ ${build_dir}/example : ${example_src:%.c
$(strip ${LINK.EXE})
-include ${example_src:%.cc=${build_dir}/%.d}
@@ -60,7 +60,7 @@ Description: Fix unused variable
${build_dir}/%.a : Makefile | ${build_dir}
$(strip ${LINK.A})
-@@ -122,6 +133,13 @@
+@@ -122,6 +133,13 @@ ${build_dir}/%.o : src/%.cc Makefile | $
${build_dir}/%.o : src/%.c Makefile | ${build_dir}
$(strip ${COMPILE.C})
@@ -74,7 +74,7 @@ Description: Fix unused variable
%.S : cppflags += ${cppflags.tbb} ${cppflags.moodycamel} ${cppflags.xenium}
%.S : src/%.cc Makefile | ${build_dir}
$(strip ${COMPILE.S})
-@@ -132,11 +150,14 @@
+@@ -132,11 +150,14 @@ ${build_dir}/%.o : src/%.c Makefile | ${
${build_dir} :
mkdir -p $@
=====================================
debian/rules
=====================================
@@ -26,7 +26,7 @@ override_dh_install:
--movedev include usr \
$$(find . -name "*.so")
-# FIXME: No idea why dh_strip fails with
+# FIXME: No idea why dh_strip fails with
#strip: debian/libatomic-queue-dev/usr/lib/x86_64-linux-gnu/libatomic_queue.a: sorry: copying thin archives is not currently supported: invalid operation
#dh_strip: error: strip --strip-debug --remove-section=.comment --remove-section=.note --enable-deterministic-archives -R .gnu.lto_\* -R .gnu.debuglto_\* -N __gnu_lto_slim -N __gnu_lto_v1 debian/libatomic-queue-dev/usr/lib/x86_64-linux-gnu/libatomic_queue.a returned exit code 1
#dh_strip: error: Aborting due to earlier error
=====================================
debian/upstream/metadata
=====================================
@@ -0,0 +1,3 @@
+---
+Bug-Database: https://github.com/max0x7ba/atomic_queue/issues
+Bug-Submit: https://github.com/max0x7ba/atomic_queue/issues/new
=====================================
include/atomic_queue/atomic_queue.h
=====================================
@@ -211,7 +211,7 @@ protected:
else {
for(;;) {
unsigned char expected = STORED;
- if(ATOMIC_QUEUE_LIKELY(state.compare_exchange_strong(expected, LOADING, X, X))) {
+ if(ATOMIC_QUEUE_LIKELY(state.compare_exchange_strong(expected, LOADING, A, X))) {
T element{std::move(q_element)};
state.store(EMPTY, R);
return element;
@@ -236,7 +236,7 @@ protected:
else {
for(;;) {
unsigned char expected = EMPTY;
- if(ATOMIC_QUEUE_LIKELY(state.compare_exchange_strong(expected, STORING, X, X))) {
+ if(ATOMIC_QUEUE_LIKELY(state.compare_exchange_strong(expected, STORING, A, X))) {
q_element = std::forward<U>(element);
state.store(STORED, R);
return;
=====================================
src/example.cc
=====================================
@@ -9,20 +9,19 @@
#include <iostream>
int main() {
- int constexpr PRODUCERS = 1;
- int constexpr CONSUMERS = 2;
- unsigned constexpr CAPACITY = 1024;
- unsigned constexpr N = 1000000;
+ int constexpr PRODUCERS = 1; // Number of producer threads.
+ int constexpr CONSUMERS = 2; // Number of consumer threads.
+ unsigned constexpr N = 1000000; // Pass this many elements from producers to consumers.
+ unsigned constexpr CAPACITY = 1024; // Queue capacity. Since there are more consumers than producers this doesn't have to be large.
- using Element = uint32_t;
- Element constexpr NIL = static_cast<Element>(-1);
+ using Element = uint32_t; // Queue element type.
+ Element constexpr NIL = static_cast<Element>(-1); // Atomic elements require a special value that cannot be pushed/popped.
+ using Queue = atomic_queue::AtomicQueueB<Element, std::allocator<Element>, NIL>; // Use heap-allocated buffer.
- using Queue = atomic_queue::AtomicQueueB<Element, std::allocator<Element>, NIL>;
-
- // Create a queue shared between producers and consumers.
+ // Create a queue object shared between producers and consumers.
Queue q{CAPACITY};
- // Start consumers.
+ // Start the consumers.
uint64_t results[CONSUMERS];
std::thread consumers[CONSUMERS];
for(int i = 0; i < CONSUMERS; ++i)
@@ -33,7 +32,7 @@ int main() {
r = sum;
});
- // Start producers.
+ // Start the producers.
std::thread producers[PRODUCERS];
for(int i = 0; i < PRODUCERS; ++i)
producers[i] = std::thread([&q]() {
@@ -45,14 +44,14 @@ int main() {
for(auto& t : producers)
t.join();
- // Stop consumers.
+ // Tell each consumer to complete and terminate.
for(int i = CONSUMERS; i--;)
q.push(0);
// Wait till consumers complete and terminate.
for(auto& t : consumers)
t.join();
- // Verify the results.
+ // Verify that each message was received exactly by one consumer only.
uint64_t result = 0;
for(auto& r : results) {
result += r;
View it on GitLab: https://salsa.debian.org/med-team/libatomic-queue/-/compare/69d9f923770b470ee6090b979843e4045157311f...60514da9e593e041dae84fff20e8ac8d023af79d
--
View it on GitLab: https://salsa.debian.org/med-team/libatomic-queue/-/compare/69d9f923770b470ee6090b979843e4045157311f...60514da9e593e041dae84fff20e8ac8d023af79d
You're receiving this email because of your account on salsa.debian.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/debian-med-commit/attachments/20201205/c0cc85ed/attachment-0001.html>
More information about the debian-med-commit
mailing list