[Nut-upsuser] Supporting a DIY UPS with minimal effort but maximum gain

[Jim Klimov]

Hello, and welcome (again) :)

> ...That [driver] executable will talk to the hardware in whatever way it
sees fit, and to nut over the UNIX socket/pipe. It does sound
architecturally feasible, at least, no?

This is more or less what NUT drivers themselves do, so yes - quite
feasible. Architecturally, NUT drivers, data server (upsd) and various
clients are separated for this reason of flexibility and whatnot (they can
die and be recycled independently, developed incompatibly, block on I/O
without impacting others, etc.). There were discussions about maybe making
some umbrella driver core program so the actual drivers would be threads in
it, but it is not likely to happen - especially for drivers of unrelated
hardware. Many threads of, say, `snmp-ups` monitoring different devices?
Maybe it makes sense, maybe not.

Arnaud Quette also proposed a "SmartNUT" project geared towards IoT
use-cases, and based on experience with a HAL back-end that was scrapped
around NUT 2.6.5 as GNOME3 deprecated HAL (note that GNOME2 clones like
MATE are still popular, so there may be value in reviving it too). Such
solutions can use a different communications backend than an Unix socket
talking to `upsd` - e.g. to have NUT drivers present an UPS directly as a
battery device on an OS bus. For some use-cases this can save some bytes in
storage and RAM footprint, and CPU cycles to pass bytes around within a
single device, especially of interest in embedded cases where resource
constraints still matter a lot.

I know some further projects based on NUT also have their own drivers so
sometimes there's a bit of constructive friction to get them ported to new
capabilities of NUT (and to keep the Unix and networked protocols and
library APIs as stable as feasible, or changes well announced and
documented). Probably one major difference, potentially, is that NUT
drivers share a large part of codebase with `common/*` code and a
`drivers/main.c` file, so they have same facilities to parse
configurations, use a data state model and announce it to the data server
(over Unix protocol), same ways of enabling debugging and whatnot.

The "third-party" drivers that I knew of were closer to what you proposed
as a "script or something", in that they did talk the NUT local socket/pipe
protocol to `upsd`, but did not use any NUT codebase. Not sure why; best
guess could be maybe to be licensed independently (perhaps using further
proprietary or differently licensed code that they can not risk linking
with GPL), or maybe to require less maintenance as the NUT codebase changes
and they would have to refactor and recompile regularly to stay on top of
things.

On the performance side, remember that things like opening/closing file
descriptors or spawning processes can be quite costly on some platforms
(many security checks involved), so you'd want to avoid e.g. shell
scripting running in a tight loop to monitor the devices, lest the driver
become a primary load for the (smallish) computer, and prefer some compiled
or pre-interpreted language (perl/python included). For a relaxed loop or
for prototyping even that shell may be okay - at least, often is easier to
tinker in the field, to an extent.

As a NUT maintainer, I guess such external drivers make sense for users who
roll their own devices and are also their consumer base (whether
individuals or corporations). If however this is something worth sharing,
maybe as a new "reference" driver (or small addition to some existing
driver like the arduino-hid subdriver) for a nifty project using your new
and also published micro-controller like here for the DIY-UPS cases, it
makes sense to upstream that driver into NUT and let it evolve and be
maintained independently of yourself. I've had tons of PRs to other
projects with the primary reasons being to share, to take the maintenance
burden off myself (and avoid regular re-merge and rebuild to have both
feature sets in my deployments), and of course to have my contributions
reviewed and stupid (or un-trivially hidden) mistakes weeded out :)

Hope this helps,
Jim Klimov



On Sun, May 19, 2024 at 8:06 PM Kiril Zyapkov <kiril.zyapkov at gmail.com>
wrote:

> Hello all,
>
> Now that I am subscribed to the list I can reply at last :)
>
> Kelly, thank you for the comprehensive write-up and pointers to sources
> and the Arduino lib and example. This took me into a rabbit hole and I've
> spent entirely too many hours browsing through the NUT sources and
> USB/HID/PowerDevice specs.
>
> Jim, thanks for the comprehensive write-up and all the work you do on NUT.
> This project has definitely accumulated heaps of wisdom in-between the
> source lines over the years. The picture is much clearer now, and the scope
> of the effort also. Not an easy feat! I wish I had cycles to spare.
>
> For my project I'm probably going for the USB HID approach.
>
> Just one final question -- is it possible to allow running a driver as an
> external executable? That way we can shift the whole problem outside of
> NUT. Yes, probing and scanning won't work, it will not be possible to
> ensure support on all platforms, drivers implemented in scripts will drag
> their own set of dependencies and what not. I am not sure how nut runs the
> driver process and who takes care of its lifetime, but if it were possible
> to use just any executable as a driver via some small stub the entire
> problem gets shifted elsewhere. That executable will talk to the hardware
> in whatever way it sees fit, and to nut over the UNIX socket/pipe. It does
> sound architecturally feasible, at least, no?
>
> Cheers,
> Kiril
>
>
> On Sat, 18 May 2024 at 20:09, Jim Klimov <jimklimov+nut at gmail.com> wrote:
>
>> Hello all,
>>
>>   I think there was a very good reply about an Arduino-based controller
>> for a DIY UPS here. The project you posted to, with an Arduino presenting
>> as a Megatec protocol server, also seems interesting.
>>
>>   Here I'd like to reply to one point not covered before - DMF. As a
>> short and quick reply - unfortunately no, you can not use it with stock
>> upstream NUT at the moment, and not for USB when customized code is
>> involved. That said, it is a long-term hope that one day this would be
>> possible. Each recent NUT release milestone had some work that chipped away
>> the technical barriers to including DMF into the main project.
>>
>>   For a longer version: DMF was designed and developed, fairly well
>> tested and then a bit abandoned in the 42ITy fork of NUT, which went back
>> to using upstream NUT for their appliances. The developed code base however
>> for a long time acts as the source of backports for features and vendor
>> device nuances into upstream NUT to benefit everyone. That codebase is
>> available in the FTY branch, including DMF along with many other changes
>> (thousands of commits) so separating them into cleanly visible and
>> revisable increments, and digesting in the upstream code, aligning with
>> protocol and other changes that happened over the years, style and
>> tests/quality gates, is quite an effort. Conversely, merging back the
>> upstream NUT into the FTY codebase (to achieve green CI survival) was an
>> effort that took a few years, since the baseline was NUT v2.7.4 and
>> codebases diverged significantly in some spots - so there were quite a few
>> warnings found that needed addressing. Currently the FTY branch is
>> relatively recently synchronised with NUT, at least buildable and passes CI
>> tests, including the DMF aspect. And since the codebases converged again
>> recently - now as stuff gets upstreamed here and there, a mere resync and
>> `git diff FTY..master` helps see how much is left, what commits were missed
>> or typos added, and what can be picked up next :)
>>
>>   DMF per se is an effort to separate the binary driver from data
>> mappings, in cases where logic remains the same and just the data points
>> differ from device to device (e.g. SNMP OIDs to query for information about
>> different aspects of UPS or ePDU state). This should well apply to USB HID
>> and maybe to nutdrv_qx; a recently added concept of `hwmon` driver (talking
>> to sysfs nodes) is also a good candidate.
>>
>>   Initial development in the 42ITy project provided a separate library
>> for DMF general foundations, and build-time changes to the snmp-ups driver
>> (and nut-scanner snmp mode) to use mappings from XML files rather than
>> built into the driver binary once and forever until a rebuild. This allows
>> for smaller binaries on one hand, and for field-maintainable mappings (edit
>> XML, add another) to support new devices without changing NUT packages and
>> programs (of course, PRs would be welcome to add such changes to the
>> upstream library to be included in future releases). In fact, the few
>> differences for the core driver are hidden by `ifdef` (so there are two
>> binary builds, with built-in mappings and with DMF support), and the
>> `nut-scanner` tool is dually-capable in the same build. In fact, to test
>> the theory, NUT codebase in the branch provides scripts that convert
>> `*-mib.c` tables into equivalent DMF XML files, so a copy of `snmp-ups-dmf`
>> driver program with the bunch of text resource files is equivalent out of
>> the box to a monolithic `snmp-ups` binary built from same codebase, but
>> more extensible in the field afterwards.
>>
>>   Part of "abandonment" of this approach was the uncertainty of how
>> correct the big wad of new code it is (hard to guarantee with that older
>> NUT baseline which naturally had thousands of warnings so lots of
>> diagnostic noise) and suspicions that it had memory leaks so maybe had
>> issues with long-term stability (not proven to be definitely the root cause
>> of some issues, as far as I am aware). Those were issues entangled with
>> semi-commercial project priorities (rush to market - pick the lowest
>> hanging fruits, backlog the rest) so it was left on a backburner, awaiting
>> a revival in upstream NUT.
>>
>>   Another part was more technical, a sort of stalemate in design: many
>> mapping tables in NUT involve data conversions (e.g. date formats,
>> temperature units, integer milliVolts to floating-point Volts, text labels
>> for numeric enum values, etc.) for data transfers from a device or writes
>> back to it. In the current C codebase (*-mib.c, *-hid.c) there are helper
>> methods to which we can point from the tables. An equivalent for DMF, with
>> mappings conveyed by text files, involved adding LUA scripts with a LUA-C
>> bridge to pass the data from binary driver to scripting context and back.
>> This worked reasonably well; the problem was having two potentially
>> different implementations of the helpers (C and LUA) for mappings provided
>> with NUT - and no good way (at least back then) to either compare that they
>> behave identically or to eliminate one (likely the native C... but then add
>> LUA as a requirement for NUT builds everywhere and provide the LUA
>> implementations of helpers via files or built-in strings). This aspect
>> stalled in discussions - not an insurmountable problem, but no single
>> apparently good solution (at least fitting the commercial project's
>> constraints) was finalized.
>>
>>   Probably a good way forward here is to track the C helper methods in
>> the NUT codebase in separate files (or wrapped by macros) so that scripting
>> can identify their existence. Then we can make sure the same set of code is
>> available in LUA, and that all of these are covered by unit tests. This
>> part seems fairly easy to automate, and can lead to a decent library of
>> helper implementations provided by NUT with DMF-capable releases. Probably
>> some duplicate efforts can also be identified this way. Note also that the
>> "problem" originally concerns the subdrivers provided by NUT as C mapping
>> tables *and* the requested ability to provide functionally identical DMF
>> clones. Discrepancies of C and LUA behaviors are not a problem for drivers
>> where DMF resource files are the only code base that defines the device
>> interactions.
>>
>>   On the other hand, if we end up with a well defined library of C
>> implementations of popular/common helpers for the mappings, maybe it would
>> be viable to (re-)use known C method names from LUA with minimal one-line
>> scripts (or specially processed XML tags) pointing to the C library methods
>> built into a driverbinary right away - also simplifying the field support
>> for new devices that rely on same concepts as something that was already
>> handled earlier.
>>
>>   To summarize - currently we have a branch (which is not yet upstreamed,
>> but finally is relatively close to that) with the core DMF library that can
>> store and process the mappings and LUA helpers, and a consumer of that
>> library for SNMP mapping purposes, further used by snmp-ups driver and
>> nut-scanner tool, and scripts that convert C mapping tables to DMF XML
>> files. There are some nuances about ensuring and testing their equivalence.
>> Once the approach is deemed unambiguously good, further sets of consumers
>> can be added for several other categories of drivers where logic is cleanly
>> separable from data mappings - most of these are largely C tables already.
>>
>> Hope this helps,
>> Jim Klimov
>>
>>
>> On Thu, May 16, 2024 at 12:27 AM Kiril Zyapkov via Nut-upsuser <
>> nut-upsuser at alioth-lists.debian.net> wrote:
>>
>>> Hello,
>>>
>>> I found out about NUT just days ago while searching for a solution for
>>> my home setup. After some digging through the interwebs, I come to you with
>>> questions.
>>>
>>> I'm putting together a DIY 12V UPS, very similar to what this guy did:
>>>
>>> [1]
>>> https://baldpenguin.blogspot.com/2015/10/diy-12v-ups-for-home-network-equipment.html
>>>
>>> The objective is to keep a bunch of mini PCs and network gear online for
>>> as long as the battery lasts and then provide a mechanism for a graceful
>>> shutdown of my NAS and other appliances for which cutting power would not
>>> be healthy. The project above is missing the "connected" part. I want to
>>> get mine to play with NUT nicely. Other prior art is this project:
>>>
>>> [2] https://github.com/xm381/Raspberry-Pi-UPS
>>>
>>> Mentioned in a previous thread here:
>>>
>>> [3]
>>> https://alioth-lists.debian.net/pipermail/nut-upsuser/2018-August/011198.html
>>>
>>> A valid approach -- emulates an existing protocol on an arduino.
>>>
>>> Are there other similar projects that you know of? I found plenty of
>>> "DIY UPS" projects, but none were "smart".
>>>
>>> I am able to put together firmware for some micro which will take care
>>> of measuring voltages, currents, possibly also turn on/off loads, serial or
>>> USB or IP are options. Not sure yet what hardware features I'll put
>>> together, but this depends somewhat on the approach for getting this thing
>>> integrated with NUT. PSUs and batteries are already on the way, and my junk
>>> drawers have most other parts I may need.
>>>
>>> So, options found so far:
>>>
>>> * Use genericups. Least favorite option, very limited features
>>>
>>> * Use the same approach as [2]. If I were to go that route -- which is
>>> the best protocol to pick for emulation? I'm looking for something simple,
>>> extensible/flexible and well-documented.
>>>
>>> But what I really wish was possible was the ability to describe my
>>> device in some format, feed it to a generic driver in NUT and profit. I see
>>> some efforts have been made in this direction, most notably:
>>>
>>> [4] https://github.com/networkupstools/nut/wiki/Data-Mapping-File-(DMF)
>>>
>>> What is the state there? Is it usable for USB HID? Or, how hard would it
>>> be to make it usable? Even a modbus description will do -- implementing the
>>> modbus server (yes, server, I'm being politically-correct) over serial or
>>> even TCP is easy, if only there was a way to dump a CSV with register
>>> descriptions in some magical driver...
>>>
>>> And yet another approach which comes to mind is to implement my driver
>>> as an external executable. This may be completely unfeasible and stupid,
>>> and please let me know if it is. But, from what I gather, drivers run in
>>> their own process and talk to the daemon via a UNIX socket. Why not make it
>>> possible for the driver to be just any executable, built/deployed outside
>>> of the NUT codebase? The socket protocol seems simple enough, and this will
>>> allow for ... creativity. It could be implemented in any language
>>> (including scripting languages) and need not depend on anything
>>> NUT-specific, other than maybe some common CLI interface and/or
>>> configuration.
>>>
>>> I'm hoping the NUT masters will have some insight. Thanks for working on
>>> this!
>>>
>>> Cheers,
>>> Kiril
>>> _______________________________________________
>>> Nut-upsuser mailing list
>>> Nut-upsuser at alioth-lists.debian.net
>>> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/nut-upsuser
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/nut-upsuser/attachments/20240519/373e0da3/attachment-0001.htm>