[Nut-upsuser] Supporting a DIY UPS with minimal effort but maximum gain

[Kiril Zyapkov]

Hello all,

Now that I am subscribed to the list I can reply at last :)

Kelly, thank you for the comprehensive write-up and pointers to sources and
the Arduino lib and example. This took me into a rabbit hole and I've spent
entirely too many hours browsing through the NUT sources and
USB/HID/PowerDevice specs.

Jim, thanks for the comprehensive write-up and all the work you do on NUT.
This project has definitely accumulated heaps of wisdom in-between the
source lines over the years. The picture is much clearer now, and the scope
of the effort also. Not an easy feat! I wish I had cycles to spare.

For my project I'm probably going for the USB HID approach.

Just one final question -- is it possible to allow running a driver as an
external executable? That way we can shift the whole problem outside of
NUT. Yes, probing and scanning won't work, it will not be possible to
ensure support on all platforms, drivers implemented in scripts will drag
their own set of dependencies and what not. I am not sure how nut runs the
driver process and who takes care of its lifetime, but if it were possible
to use just any executable as a driver via some small stub the entire
problem gets shifted elsewhere. That executable will talk to the hardware
in whatever way it sees fit, and to nut over the UNIX socket/pipe. It does
sound architecturally feasible, at least, no?

Cheers,
Kiril


On Sat, 18 May 2024 at 20:09, Jim Klimov <jimklimov+nut at gmail.com> wrote:

> Hello all,
>
>   I think there was a very good reply about an Arduino-based controller
> for a DIY UPS here. The project you posted to, with an Arduino presenting
> as a Megatec protocol server, also seems interesting.
>
>   Here I'd like to reply to one point not covered before - DMF. As a short
> and quick reply - unfortunately no, you can not use it with stock upstream
> NUT at the moment, and not for USB when customized code is involved. That
> said, it is a long-term hope that one day this would be possible. Each
> recent NUT release milestone had some work that chipped away the technical
> barriers to including DMF into the main project.
>
>   For a longer version: DMF was designed and developed, fairly well tested
> and then a bit abandoned in the 42ITy fork of NUT, which went back to using
> upstream NUT for their appliances. The developed code base however for a
> long time acts as the source of backports for features and vendor device
> nuances into upstream NUT to benefit everyone. That codebase is available
> in the FTY branch, including DMF along with many other changes (thousands
> of commits) so separating them into cleanly visible and revisable
> increments, and digesting in the upstream code, aligning with protocol and
> other changes that happened over the years, style and tests/quality gates,
> is quite an effort. Conversely, merging back the upstream NUT into the FTY
> codebase (to achieve green CI survival) was an effort that took a few
> years, since the baseline was NUT v2.7.4 and codebases diverged
> significantly in some spots - so there were quite a few warnings found that
> needed addressing. Currently the FTY branch is relatively recently
> synchronised with NUT, at least buildable and passes CI tests, including
> the DMF aspect. And since the codebases converged again recently - now as
> stuff gets upstreamed here and there, a mere resync and `git diff
> FTY..master` helps see how much is left, what commits were missed or typos
> added, and what can be picked up next :)
>
>   DMF per se is an effort to separate the binary driver from data
> mappings, in cases where logic remains the same and just the data points
> differ from device to device (e.g. SNMP OIDs to query for information about
> different aspects of UPS or ePDU state). This should well apply to USB HID
> and maybe to nutdrv_qx; a recently added concept of `hwmon` driver (talking
> to sysfs nodes) is also a good candidate.
>
>   Initial development in the 42ITy project provided a separate library for
> DMF general foundations, and build-time changes to the snmp-ups driver (and
> nut-scanner snmp mode) to use mappings from XML files rather than built
> into the driver binary once and forever until a rebuild. This allows for
> smaller binaries on one hand, and for field-maintainable mappings (edit
> XML, add another) to support new devices without changing NUT packages and
> programs (of course, PRs would be welcome to add such changes to the
> upstream library to be included in future releases). In fact, the few
> differences for the core driver are hidden by `ifdef` (so there are two
> binary builds, with built-in mappings and with DMF support), and the
> `nut-scanner` tool is dually-capable in the same build. In fact, to test
> the theory, NUT codebase in the branch provides scripts that convert
> `*-mib.c` tables into equivalent DMF XML files, so a copy of `snmp-ups-dmf`
> driver program with the bunch of text resource files is equivalent out of
> the box to a monolithic `snmp-ups` binary built from same codebase, but
> more extensible in the field afterwards.
>
>   Part of "abandonment" of this approach was the uncertainty of how
> correct the big wad of new code it is (hard to guarantee with that older
> NUT baseline which naturally had thousands of warnings so lots of
> diagnostic noise) and suspicions that it had memory leaks so maybe had
> issues with long-term stability (not proven to be definitely the root cause
> of some issues, as far as I am aware). Those were issues entangled with
> semi-commercial project priorities (rush to market - pick the lowest
> hanging fruits, backlog the rest) so it was left on a backburner, awaiting
> a revival in upstream NUT.
>
>   Another part was more technical, a sort of stalemate in design: many
> mapping tables in NUT involve data conversions (e.g. date formats,
> temperature units, integer milliVolts to floating-point Volts, text labels
> for numeric enum values, etc.) for data transfers from a device or writes
> back to it. In the current C codebase (*-mib.c, *-hid.c) there are helper
> methods to which we can point from the tables. An equivalent for DMF, with
> mappings conveyed by text files, involved adding LUA scripts with a LUA-C
> bridge to pass the data from binary driver to scripting context and back.
> This worked reasonably well; the problem was having two potentially
> different implementations of the helpers (C and LUA) for mappings provided
> with NUT - and no good way (at least back then) to either compare that they
> behave identically or to eliminate one (likely the native C... but then add
> LUA as a requirement for NUT builds everywhere and provide the LUA
> implementations of helpers via files or built-in strings). This aspect
> stalled in discussions - not an insurmountable problem, but no single
> apparently good solution (at least fitting the commercial project's
> constraints) was finalized.
>
>   Probably a good way forward here is to track the C helper methods in the
> NUT codebase in separate files (or wrapped by macros) so that scripting can
> identify their existence. Then we can make sure the same set of code is
> available in LUA, and that all of these are covered by unit tests. This
> part seems fairly easy to automate, and can lead to a decent library of
> helper implementations provided by NUT with DMF-capable releases. Probably
> some duplicate efforts can also be identified this way. Note also that the
> "problem" originally concerns the subdrivers provided by NUT as C mapping
> tables *and* the requested ability to provide functionally identical DMF
> clones. Discrepancies of C and LUA behaviors are not a problem for drivers
> where DMF resource files are the only code base that defines the device
> interactions.
>
>   On the other hand, if we end up with a well defined library of C
> implementations of popular/common helpers for the mappings, maybe it would
> be viable to (re-)use known C method names from LUA with minimal one-line
> scripts (or specially processed XML tags) pointing to the C library methods
> built into a driverbinary right away - also simplifying the field support
> for new devices that rely on same concepts as something that was already
> handled earlier.
>
>   To summarize - currently we have a branch (which is not yet upstreamed,
> but finally is relatively close to that) with the core DMF library that can
> store and process the mappings and LUA helpers, and a consumer of that
> library for SNMP mapping purposes, further used by snmp-ups driver and
> nut-scanner tool, and scripts that convert C mapping tables to DMF XML
> files. There are some nuances about ensuring and testing their equivalence.
> Once the approach is deemed unambiguously good, further sets of consumers
> can be added for several other categories of drivers where logic is cleanly
> separable from data mappings - most of these are largely C tables already.
>
> Hope this helps,
> Jim Klimov
>
>
> On Thu, May 16, 2024 at 12:27 AM Kiril Zyapkov via Nut-upsuser <
> nut-upsuser at alioth-lists.debian.net> wrote:
>
>> Hello,
>>
>> I found out about NUT just days ago while searching for a solution for my
>> home setup. After some digging through the interwebs, I come to you with
>> questions.
>>
>> I'm putting together a DIY 12V UPS, very similar to what this guy did:
>>
>> [1]
>> https://baldpenguin.blogspot.com/2015/10/diy-12v-ups-for-home-network-equipment.html
>>
>> The objective is to keep a bunch of mini PCs and network gear online for
>> as long as the battery lasts and then provide a mechanism for a graceful
>> shutdown of my NAS and other appliances for which cutting power would not
>> be healthy. The project above is missing the "connected" part. I want to
>> get mine to play with NUT nicely. Other prior art is this project:
>>
>> [2] https://github.com/xm381/Raspberry-Pi-UPS
>>
>> Mentioned in a previous thread here:
>>
>> [3]
>> https://alioth-lists.debian.net/pipermail/nut-upsuser/2018-August/011198.html
>>
>> A valid approach -- emulates an existing protocol on an arduino.
>>
>> Are there other similar projects that you know of? I found plenty of "DIY
>> UPS" projects, but none were "smart".
>>
>> I am able to put together firmware for some micro which will take care of
>> measuring voltages, currents, possibly also turn on/off loads, serial or
>> USB or IP are options. Not sure yet what hardware features I'll put
>> together, but this depends somewhat on the approach for getting this thing
>> integrated with NUT. PSUs and batteries are already on the way, and my junk
>> drawers have most other parts I may need.
>>
>> So, options found so far:
>>
>> * Use genericups. Least favorite option, very limited features
>>
>> * Use the same approach as [2]. If I were to go that route -- which is
>> the best protocol to pick for emulation? I'm looking for something simple,
>> extensible/flexible and well-documented.
>>
>> But what I really wish was possible was the ability to describe my device
>> in some format, feed it to a generic driver in NUT and profit. I see some
>> efforts have been made in this direction, most notably:
>>
>> [4] https://github.com/networkupstools/nut/wiki/Data-Mapping-File-(DMF)
>>
>> What is the state there? Is it usable for USB HID? Or, how hard would it
>> be to make it usable? Even a modbus description will do -- implementing the
>> modbus server (yes, server, I'm being politically-correct) over serial or
>> even TCP is easy, if only there was a way to dump a CSV with register
>> descriptions in some magical driver...
>>
>> And yet another approach which comes to mind is to implement my driver as
>> an external executable. This may be completely unfeasible and stupid, and
>> please let me know if it is. But, from what I gather, drivers run in their
>> own process and talk to the daemon via a UNIX socket. Why not make it
>> possible for the driver to be just any executable, built/deployed outside
>> of the NUT codebase? The socket protocol seems simple enough, and this will
>> allow for ... creativity. It could be implemented in any language
>> (including scripting languages) and need not depend on anything
>> NUT-specific, other than maybe some common CLI interface and/or
>> configuration.
>>
>> I'm hoping the NUT masters will have some insight. Thanks for working on
>> this!
>>
>> Cheers,
>> Kiril
>> _______________________________________________
>> Nut-upsuser mailing list
>> Nut-upsuser at alioth-lists.debian.net
>> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/nut-upsuser
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://alioth-lists.debian.net/pipermail/nut-upsuser/attachments/20240519/6121861c/attachment-0001.htm>