[Nut-upsuser] Supporting a DIY UPS with minimal effort but maximum gain

[Jim Klimov]

Hello all,

  I think there was a very good reply about an Arduino-based controller for
a DIY UPS here. The project you posted to, with an Arduino presenting as a
Megatec protocol server, also seems interesting.

  Here I'd like to reply to one point not covered before - DMF. As a short
and quick reply - unfortunately no, you can not use it with stock upstream
NUT at the moment, and not for USB when customized code is involved. That
said, it is a long-term hope that one day this would be possible. Each
recent NUT release milestone had some work that chipped away the technical
barriers to including DMF into the main project.

  For a longer version: DMF was designed and developed, fairly well tested
and then a bit abandoned in the 42ITy fork of NUT, which went back to using
upstream NUT for their appliances. The developed code base however for a
long time acts as the source of backports for features and vendor device
nuances into upstream NUT to benefit everyone. That codebase is available
in the FTY branch, including DMF along with many other changes (thousands
of commits) so separating them into cleanly visible and revisable
increments, and digesting in the upstream code, aligning with protocol and
other changes that happened over the years, style and tests/quality gates,
is quite an effort. Conversely, merging back the upstream NUT into the FTY
codebase (to achieve green CI survival) was an effort that took a few
years, since the baseline was NUT v2.7.4 and codebases diverged
significantly in some spots - so there were quite a few warnings found that
needed addressing. Currently the FTY branch is relatively recently
synchronised with NUT, at least buildable and passes CI tests, including
the DMF aspect. And since the codebases converged again recently - now as
stuff gets upstreamed here and there, a mere resync and `git diff
FTY..master` helps see how much is left, what commits were missed or typos
added, and what can be picked up next :)

  DMF per se is an effort to separate the binary driver from data mappings,
in cases where logic remains the same and just the data points differ from
device to device (e.g. SNMP OIDs to query for information about different
aspects of UPS or ePDU state). This should well apply to USB HID and maybe
to nutdrv_qx; a recently added concept of `hwmon` driver (talking to sysfs
nodes) is also a good candidate.

  Initial development in the 42ITy project provided a separate library for
DMF general foundations, and build-time changes to the snmp-ups driver (and
nut-scanner snmp mode) to use mappings from XML files rather than built
into the driver binary once and forever until a rebuild. This allows for
smaller binaries on one hand, and for field-maintainable mappings (edit
XML, add another) to support new devices without changing NUT packages and
programs (of course, PRs would be welcome to add such changes to the
upstream library to be included in future releases). In fact, the few
differences for the core driver are hidden by `ifdef` (so there are two
binary builds, with built-in mappings and with DMF support), and the
`nut-scanner` tool is dually-capable in the same build. In fact, to test
the theory, NUT codebase in the branch provides scripts that convert
`*-mib.c` tables into equivalent DMF XML files, so a copy of `snmp-ups-dmf`
driver program with the bunch of text resource files is equivalent out of
the box to a monolithic `snmp-ups` binary built from same codebase, but
more extensible in the field afterwards.

  Part of "abandonment" of this approach was the uncertainty of how correct
the big wad of new code it is (hard to guarantee with that older NUT
baseline which naturally had thousands of warnings so lots of diagnostic
noise) and suspicions that it had memory leaks so maybe had issues with
long-term stability (not proven to be definitely the root cause of some
issues, as far as I am aware). Those were issues entangled with
semi-commercial project priorities (rush to market - pick the lowest
hanging fruits, backlog the rest) so it was left on a backburner, awaiting
a revival in upstream NUT.

  Another part was more technical, a sort of stalemate in design: many
mapping tables in NUT involve data conversions (e.g. date formats,
temperature units, integer milliVolts to floating-point Volts, text labels
for numeric enum values, etc.) for data transfers from a device or writes
back to it. In the current C codebase (*-mib.c, *-hid.c) there are helper
methods to which we can point from the tables. An equivalent for DMF, with
mappings conveyed by text files, involved adding LUA scripts with a LUA-C
bridge to pass the data from binary driver to scripting context and back.
This worked reasonably well; the problem was having two potentially
different implementations of the helpers (C and LUA) for mappings provided
with NUT - and no good way (at least back then) to either compare that they
behave identically or to eliminate one (likely the native C... but then add
LUA as a requirement for NUT builds everywhere and provide the LUA
implementations of helpers via files or built-in strings). This aspect
stalled in discussions - not an insurmountable problem, but no single
apparently good solution (at least fitting the commercial project's
constraints) was finalized.

  Probably a good way forward here is to track the C helper methods in the
NUT codebase in separate files (or wrapped by macros) so that scripting can
identify their existence. Then we can make sure the same set of code is
available in LUA, and that all of these are covered by unit tests. This
part seems fairly easy to automate, and can lead to a decent library of
helper implementations provided by NUT with DMF-capable releases. Probably
some duplicate efforts can also be identified this way. Note also that the
"problem" originally concerns the subdrivers provided by NUT as C mapping
tables *and* the requested ability to provide functionally identical DMF
clones. Discrepancies of C and LUA behaviors are not a problem for drivers
where DMF resource files are the only code base that defines the device
interactions.

  On the other hand, if we end up with a well defined library of C
implementations of popular/common helpers for the mappings, maybe it would
be viable to (re-)use known C method names from LUA with minimal one-line
scripts (or specially processed XML tags) pointing to the C library methods
built into a driverbinary right away - also simplifying the field support
for new devices that rely on same concepts as something that was already
handled earlier.

  To summarize - currently we have a branch (which is not yet upstreamed,
but finally is relatively close to that) with the core DMF library that can
store and process the mappings and LUA helpers, and a consumer of that
library for SNMP mapping purposes, further used by snmp-ups driver and
nut-scanner tool, and scripts that convert C mapping tables to DMF XML
files. There are some nuances about ensuring and testing their equivalence.
Once the approach is deemed unambiguously good, further sets of consumers
can be added for several other categories of drivers where logic is cleanly
separable from data mappings - most of these are largely C tables already.

Hope this helps,
Jim Klimov


On Thu, May 16, 2024 at 12:27 AM Kiril Zyapkov via Nut-upsuser <
nut-upsuser at alioth-lists.debian.net> wrote:

> Hello,
>
> I found out about NUT just days ago while searching for a solution for my
> home setup. After some digging through the interwebs, I come to you with
> questions.
>
> I'm putting together a DIY 12V UPS, very similar to what this guy did:
>
> [1]
> https://baldpenguin.blogspot.com/2015/10/diy-12v-ups-for-home-network-equipment.html
>
> The objective is to keep a bunch of mini PCs and network gear online for
> as long as the battery lasts and then provide a mechanism for a graceful
> shutdown of my NAS and other appliances for which cutting power would not
> be healthy. The project above is missing the "connected" part. I want to
> get mine to play with NUT nicely. Other prior art is this project:
>
> [2] https://github.com/xm381/Raspberry-Pi-UPS
>
> Mentioned in a previous thread here:
>
> [3]
> https://alioth-lists.debian.net/pipermail/nut-upsuser/2018-August/011198.html
>
> A valid approach -- emulates an existing protocol on an arduino.
>
> Are there other similar projects that you know of? I found plenty of "DIY
> UPS" projects, but none were "smart".
>
> I am able to put together firmware for some micro which will take care of
> measuring voltages, currents, possibly also turn on/off loads, serial or
> USB or IP are options. Not sure yet what hardware features I'll put
> together, but this depends somewhat on the approach for getting this thing
> integrated with NUT. PSUs and batteries are already on the way, and my junk
> drawers have most other parts I may need.
>
> So, options found so far:
>
> * Use genericups. Least favorite option, very limited features
>
> * Use the same approach as [2]. If I were to go that route -- which is the
> best protocol to pick for emulation? I'm looking for something simple,
> extensible/flexible and well-documented.
>
> But what I really wish was possible was the ability to describe my device
> in some format, feed it to a generic driver in NUT and profit. I see some
> efforts have been made in this direction, most notably:
>
> [4] https://github.com/networkupstools/nut/wiki/Data-Mapping-File-(DMF)
>
> What is the state there? Is it usable for USB HID? Or, how hard would it
> be to make it usable? Even a modbus description will do -- implementing the
> modbus server (yes, server, I'm being politically-correct) over serial or
> even TCP is easy, if only there was a way to dump a CSV with register
> descriptions in some magical driver...
>
> And yet another approach which comes to mind is to implement my driver as
> an external executable. This may be completely unfeasible and stupid, and
> please let me know if it is. But, from what I gather, drivers run in their
> own process and talk to the daemon via a UNIX socket. Why not make it
> possible for the driver to be just any executable, built/deployed outside
> of the NUT codebase? The socket protocol seems simple enough, and this will
> allow for ... creativity. It could be implemented in any language
> (including scripting languages) and need not depend on anything
> NUT-specific, other than maybe some common CLI interface and/or
> configuration.
>
> I'm hoping the NUT masters will have some insight. Thanks for working on
> this!
>
> Cheers,
> Kiril
> _______________________________________________
> Nut-upsuser mailing list
> Nut-upsuser at alioth-lists.debian.net
> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/nut-upsuser
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/nut-upsuser/attachments/20240518/622461d3/attachment.htm>