From dr at jones.dk Tue Jan 14 11:55:13 2025 From: dr at jones.dk (Jonas Smedegaard) Date: Tue, 14 Jan 2025 12:55:13 +0100 Subject: [Blend-tinker-devel] wiki.d.o on a git-backed engine Message-ID: <173685571374.1248275.4111579148719832663@cairon.jones.dk> [moved from thread at debian-devel at lists.debian.org] Quoting Serafeim (Serafi) Zanikolas (2025-01-13 22:06:01) > On Mon Jan 13, 2025 at 12:45 AM CET, Jonas Smedegaard wrote: > > Quoting Serafeim (Serafi) Zanikolas (2025-01-12 21:54:58) > > > what would be truly amazing, imho, would be the whole wiki on git. that'd allow > > > for mass-updates, and reusing one's code (salsa) workflows for documentation > > > > For those not only wishing that others made that happen, but consider > > scratching that particular itch themselves, I am happy to chat with you > > more detailed about what that involves, and how it is a quite exciting > > challenge - so exciting, in fact, that Debian has had several attempts > > at doing a migration over the years, but none so far have succeeded. > [..] > > But let's not hijack any further this thread about welcoming newcomers, > > let's create a new thread then :) > > > I just wanted to chime in to encourage anyone wanting to do the wiki > > modernization to get in touch with me to discuss further. > > thank you for the offer but why not have the follow up in a publicly archived > list? happy to switch to -www, if -devel is not ideal > > I think that a retrospective/postmortem of past attempts would be very valuable. > I offer to write one if you'd be okay to share all of the context with me (you > could review it before it gets published). > > also, I'd think that nailing down the requirements for a new platform and for > the content to be migrated (e.g. drop any pages that are >X years old) would be > an important prerequisite for any technical work. has this been done in previous > migration attempts? to start with, do we have "rough consensus" that a > git-backed wiki would be preferable? # Wiki I guess we can agree that we want to continue to have a [quick-web], i.e. a collaborative collection of concept pages quickly user-editable and revision-controlled. I mention this because wiki has been used more broadly for systems less collaborative or less user-oriented or not revision-controlled. Do we agree that the above are requirements? Even if we think we agree, let's revisit them later, as they might turn out to need some wiggle room, e.g. making it less easy and less quick for users to edit to avoid spam. # User interface(s) One detail we might disagree on is the notion of "user", which is tied to the notion of "web": The originial WikiWikiWeb was edited via a web *browser* whereas some newer systems are edited offline and then pushed (via web or other protocols) to be read via a web browser. A few of those systems offer web-based editing as well, as an optional addon. Strictly requiring web-browser-based editing limits available options. Personally I am leaning towards separation of concerns, in that I want flexibility of choice of as fast and lightweight as possible static site generator (see below) with geeky-user editing via a git interface as required and simple-user editing via web interface as an optional addon. I envision approaching that separation through embracing the standard Micropub, and am slowly looking into implementing a [proxy] service to address that. I am lousy at coding, however, so if you think that sounds exciting then please don't wait for me to solve that challenge but consider joining me! [quick-web]: https://en.wikipedia.org/wiki/Wiki#History [Micropub]: https://www.w3.org/TR/micropub/ [proxy]: https://salsa.debian.org/tinker-team/website-edit # Markup Some esp. early wikis use markup now considered obscure, where most newer systems use some flavor of Markdown. Markdown-based wikis sometimes deviate further through custom markers, and some add metadata at the top, most commonly as YAML or TOML data. Do we want to require a certain markup, or leave that to whatever system we want for different reasons? This obviously affects the migration challenge, but perhaps not decide now, but just keep it in mind when considering system options. # Backend Most wikis store page content as files in a POSIX filesystem, or as strings in an SQL database, or as documents in a NoSQL database, as is standard for those backends, with revision control and web editing added as custom logic on top of that. Git-based wikis use a standardized revision control, with web access as custom logic on top. The use of a standardised version control system is a key feature for me, and judging from the topic of this thread also for others. Requiring git-based backend limits options, and perhaps most notably excludes the undeniably popular system Mediawiki, used e.g. for Wikipedia. Some disagree with me on that, though: https://webmasters.stackexchange.com/a/82433 # Rendering Some wikis render each page on-demand, and some render all pages ahead of time after each content edit. The latter are called static web compilers. I am a big fan of static web compilers, but whether they are more efficient obviously depends on how frequently a site is being edited versus viewed, and how expensive a full site recompilation is. # Git-backed options with web-editor A nice but potentially dated overview is here: https://paulhammant.com/2017/09/23/wikis-that-use-source-control-for-their-backing-store/ # Git-backed options without web-editor Lots of options, since most static web compilers can be scripted to use git as backend. Most popular is probably are Hugo and Jekyll. My personal favorite is Zola, less popular but like hugo among the fastest and unlike hugo uses a more common template logic. I am also pleased by Zola being written in Rust where Hugo is on go. Hugo is however already available in Debian, Zola is not (I am slowly working on it). # Migration MoinMoin use a filesystem-based backend with each page revision in a sequence-numbered subdirectory. Content markup predates Markdown and is called Creole. A plugin mechanism introduces custom markers that need caring for as well. Around 2005 I packaged the Perl modules usable for transforming wiki markup, including from MoinMoin Creole to flavors of Markdown. An attempt to migrate to Ikiwiki was investigated around 2008-2009. I vaguely remembered that effort to be lead by Frank Lin PIAT, but reading some archived emails it seems I was more convinced than him on that, so I am no longer sure what happened back then, and it was before I gained familiarity with git so I have no code lying around related to that. The best that I know of is the work by anarcat and others, maintained at https://ikiwiki.info/tips/convert_moinmoin_to_ikiwiki/ I have some unfinished code converting a relatively large ikiwiki site to Zola, which might have some use also for migrating to other systems like hugo. - Jonas -- * Jonas Smedegaard - idealist & Internet-arkitekt * Tlf.: +45 40843136 Website: http://dr.jones.dk/ * Sponsorship: https://ko-fi.com/drjones [x] quote me freely [ ] ask before reusing [ ] keep private -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 931 bytes Desc: signature URL: From dr at jones.dk Tue Jan 14 12:37:20 2025 From: dr at jones.dk (Jonas Smedegaard) Date: Tue, 14 Jan 2025 13:37:20 +0100 Subject: [Blend-tinker-devel] wiki.d.o on a git-backed engine In-Reply-To: <173685571374.1248275.4111579148719832663@cairon.jones.dk> References: <173685571374.1248275.4111579148719832663@cairon.jones.dk> Message-ID: <173685824026.1248275.1591720490347257773@cairon.jones.dk> Quoting Jonas Smedegaard (2025-01-14 12:55:13) > # Migration > > MoinMoin use a filesystem-based backend with each page revision in a > sequence-numbered subdirectory. Content markup predates Markdown and is > called Creole. A plugin mechanism introduces custom markers that need > caring for as well. > > Around 2005 I packaged the Perl modules usable for transforming wiki markup, > including from MoinMoin Creole to flavors of Markdown. Correction: Jonas Genannt began in 2005, I took over in 2007 and later added the addon packages for MoinMoin and Markdown and more: https://tracker.debian.org/pkg/libhtml-wikiconverter-perl https://tracker.debian.org/pkg/libhtml-wikiconverter-markdown-perl https://tracker.debian.org/pkg/libhtml-wikiconverter-moinmoin-perl - Jonas -- * Jonas Smedegaard - idealist & Internet-arkitekt * Tlf.: +45 40843136 Website: http://dr.jones.dk/ * Sponsorship: https://ko-fi.com/drjones [x] quote me freely [ ] ask before reusing [ ] keep private -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 931 bytes Desc: signature URL: From dr at jones.dk Tue Jan 14 12:39:42 2025 From: dr at jones.dk (Jonas Smedegaard) Date: Tue, 14 Jan 2025 13:39:42 +0100 Subject: [Blend-tinker-devel] wiki.d.o on a git-backed engine In-Reply-To: <173685824026.1248275.1591720490347257773@cairon.jones.dk> References: <173685571374.1248275.4111579148719832663@cairon.jones.dk> <173685824026.1248275.1591720490347257773@cairon.jones.dk> Message-ID: <173685838208.1248275.15055749884240054337@cairon.jones.dk> Quoting Jonas Smedegaard (2025-01-14 13:37:20) > Quoting Jonas Smedegaard (2025-01-14 12:55:13) > > # Migration > > > > MoinMoin use a filesystem-based backend with each page revision in a > > sequence-numbered subdirectory. Content markup predates Markdown and is > > called Creole. A plugin mechanism introduces custom markers that need > > caring for as well. > > > > Around 2005 I packaged the Perl modules usable for transforming wiki markup, > > including from MoinMoin Creole to flavors of Markdown. > > Correction: Jonas Genannt began in 2005, I took over in 2007 and later > added the addon packages for MoinMoin and Markdown and more: > https://tracker.debian.org/pkg/libhtml-wikiconverter-perl > https://tracker.debian.org/pkg/libhtml-wikiconverter-markdown-perl > https://tracker.debian.org/pkg/libhtml-wikiconverter-moinmoin-perl I don't deserve credit for introducing the addons either: They were part of the main library in earlier upstream releases. - Jonas -- * Jonas Smedegaard - idealist & Internet-arkitekt * Tlf.: +45 40843136 Website: http://dr.jones.dk/ * Sponsorship: https://ko-fi.com/drjones [x] quote me freely [ ] ask before reusing [ ] keep private -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 931 bytes Desc: signature URL: From ahmad at khalifa.ws Tue Jan 14 20:10:55 2025 From: ahmad at khalifa.ws (Ahmad Khalifa) Date: Tue, 14 Jan 2025 20:10:55 +0000 Subject: [Blend-tinker-devel] wiki.d.o on a git-backed engine In-Reply-To: <173685571374.1248275.4111579148719832663@cairon.jones.dk> References: <173685571374.1248275.4111579148719832663@cairon.jones.dk> Message-ID: On 14/01/2025 11:55, Jonas Smedegaard wrote: > # User interface(s) > > One detail we might disagree on is the notion of "user", which is tied > to the notion of "web": The originial WikiWikiWeb was edited via a web > *browser* whereas some newer systems are edited offline and then pushed > (via web or other protocols) to be read via a web browser. A few of > those systems offer web-based editing as well, as an optional addon. > > Strictly requiring web-browser-based editing limits available options. 1. Developer-type user (will read the code) 2. Power-user (will read the manual) 3. Contributing user (likes debian ecosystem) 4. Casual-user (uses debian OS, little time to even do a websearch, will only look at overly-SEOd pages that come up in the first page) I would say as you go from 1 to 4, the number of users go up. The biggest audience for a wiki are casual users who don't know what git is. If you can convert casual-users into contributing-user, that's great. So for me the web-editing capability is the most important feature after web-browsing :) > # Markup > > Some esp. early wikis use markup now considered obscure, where most > newer systems use some flavor of Markdown. > > Markdown-based wikis sometimes deviate further through custom markers, > and some add metadata at the top, most commonly as YAML or TOML data. > > Do we want to require a certain markup, or leave that to whatever system > we want for different reasons? > > This obviously affects the migration challenge, but perhaps not decide > now, but just keep it in mind when considering system options. Can we agree that it has to be accessible to Contributing-users? This will unintentionally bias towards markdown or mediawiki format, but that's not my intent here. > # Backend > > Most wikis store page content as files in a POSIX filesystem, or as > strings in an SQL database, or as documents in a NoSQL database, as is > standard for those backends, with revision control and web editing added > as custom logic on top of that. > > Git-based wikis use a standardized revision control, with web access as > custom logic on top. > > The use of a standardised version control system is a key feature for > me, and judging from the topic of this thread also for others. > > Requiring git-based backend limits options, and perhaps most notably > excludes the undeniably popular system Mediawiki, used e.g. for > Wikipedia. Some disagree with me on that, though: > https://webmasters.stackexchange.com/a/82433 I see git-based backend as a not-so-great feature to have. Database backends are more popular for admins. phpMyAdmin for even web-based DB-editing. If you have the same stack as Wordpress/MediaWiki and friends, then you have a much bigger pool of contributors/power users able to help. > # Rendering > > Some wikis render each page on-demand, and some render all pages ahead > of time after each content edit. The latter are called static web > compilers. > > I am a big fan of static web compilers, but whether they are more > efficient obviously depends on how frequently a site is being edited > versus viewed, and how expensive a full site recompilation is. I like SSGs, I use pelican and hugo. But neither have a web-editing interface, you have to compile it. That's Ok for developer tools. > # Migration > > MoinMoin use a filesystem-based backend with each page revision in a > sequence-numbered subdirectory. Content markup predates Markdown and is > called Creole. A plugin mechanism introduces custom markers that need > caring for as well. > > Around 2005 I packaged the Perl modules usable for transforming wiki markup, > including from MoinMoin Creole to flavors of Markdown. > > An attempt to migrate to Ikiwiki was investigated around 2008-2009. I > vaguely remembered that effort to be lead by Frank Lin PIAT, but reading > some archived emails it seems I was more convinced than him on that, so > I am no longer sure what happened back then, and it was before I gained > familiarity with git so I have no code lying around related to that. > > The best that I know of is the work by anarcat and others, maintained at > https://ikiwiki.info/tips/convert_moinmoin_to_ikiwiki/ > > I have some unfinished code converting a relatively large ikiwiki site > to Zola, which might have some use also for migrating to other systems > like hugo. When this migration happened back then, how far did it get? Stuck at 90% or much earlier? -- Regards, Ahmad From jonas at jones.dk Tue Jan 14 21:05:22 2025 From: jonas at jones.dk (Jonas Smedegaard) Date: Tue, 14 Jan 2025 22:05:22 +0100 Subject: [Blend-tinker-devel] wiki.d.o on a git-backed engine In-Reply-To: References: <173685571374.1248275.4111579148719832663@cairon.jones.dk> Message-ID: <173688872225.1248275.17393014750142863493@cairon.jones.dk> Hi Ahmad, Quoting Ahmad Khalifa (2025-01-14 21:10:55) > On 14/01/2025 11:55, Jonas Smedegaard wrote: > > # User interface(s) > > > > One detail we might disagree on is the notion of "user", which is tied > > to the notion of "web": The originial WikiWikiWeb was edited via a web > > *browser* whereas some newer systems are edited offline and then pushed > > (via web or other protocols) to be read via a web browser. A few of > > those systems offer web-based editing as well, as an optional addon. > > > > Strictly requiring web-browser-based editing limits available options. > > 1. Developer-type user (will read the code) > 2. Power-user (will read the manual) > 3. Contributing user (likes debian ecosystem) > 4. Casual-user (uses debian OS, little time to even do a websearch, will > only look at overly-SEOd pages that come up in the first page) > > I would say as you go from 1 to 4, the number of users go up. The > biggest audience for a wiki are casual users who don't know what git is. > > If you can convert casual-users into contributing-user, that's great. > So for me the web-editing capability is the most important feature after > web-browsing :) I think we can all agree that ideally all users have pleasant access. Question is not what users we want, but if we want users more than we want speed more than we want functional wiki code. I.e. a complex problem that cannot be meaningfully separated into subproblems, because deciding on one subpart (what users to cover) affects the other (what time does it take to finalize an edit in a giant wiki, what systems exist that offers a git backend, etc.). > > # Markup > > > > Some esp. early wikis use markup now considered obscure, where most > > newer systems use some flavor of Markdown. > > > > Markdown-based wikis sometimes deviate further through custom markers, > > and some add metadata at the top, most commonly as YAML or TOML data. > > > > Do we want to require a certain markup, or leave that to whatever system > > we want for different reasons? > > > > This obviously affects the migration challenge, but perhaps not decide > > now, but just keep it in mind when considering system options. > > Can we agree that it has to be accessible to Contributing-users? > This will unintentionally bias towards markdown or mediawiki format, but > that's not my intent here. Sure, but the more variables we turn into hard requirements, the less likely we can reach a functional system at all. > > # Backend > > > > Most wikis store page content as files in a POSIX filesystem, or as > > strings in an SQL database, or as documents in a NoSQL database, as is > > standard for those backends, with revision control and web editing added > > as custom logic on top of that. > > > > Git-based wikis use a standardized revision control, with web access as > > custom logic on top. > > > > The use of a standardised version control system is a key feature for > > me, and judging from the topic of this thread also for others. > > > > Requiring git-based backend limits options, and perhaps most notably > > excludes the undeniably popular system Mediawiki, used e.g. for > > Wikipedia. Some disagree with me on that, though: > > https://webmasters.stackexchange.com/a/82433 > > I see git-based backend as a not-so-great feature to have. Database > backends are more popular for admins. phpMyAdmin for even web-based > DB-editing. > If you have the same stack as Wordpress/MediaWiki and friends, then you > have a much bigger pool of contributors/power users able to help. Interesting. That was kinda the starting point of this conversation. Perhaps another conversation makes more sense, which does not start at the conversation topic "wiki.d.o on a git-backed engine". I suggest to seek conversation partners at the d-devel mailinglist for that other conversation, because the very reason I proposed to discuss it here was the shared interest in lightweight systems. Specifically, I noticed some interest in Mediawiki, and those folks is unlikely to have joined this conversation due to the (for them) off-putting conversation topic here. > > # Rendering > > > > Some wikis render each page on-demand, and some render all pages ahead > > of time after each content edit. The latter are called static web > > compilers. > > > > I am a big fan of static web compilers, but whether they are more > > efficient obviously depends on how frequently a site is being edited > > versus viewed, and how expensive a full site recompilation is. > > I like SSGs, I use pelican and hugo. But neither have a web-editing > interface, you have to compile it. That's Ok for developer tools. > > > > # Migration > > > > MoinMoin use a filesystem-based backend with each page revision in a > > sequence-numbered subdirectory. Content markup predates Markdown and is > > called Creole. A plugin mechanism introduces custom markers that need > > caring for as well. > > > > Around 2005 I packaged the Perl modules usable for transforming wiki markup, > > including from MoinMoin Creole to flavors of Markdown. > > > > An attempt to migrate to Ikiwiki was investigated around 2008-2009. I > > vaguely remembered that effort to be lead by Frank Lin PIAT, but reading > > some archived emails it seems I was more convinced than him on that, so > > I am no longer sure what happened back then, and it was before I gained > > familiarity with git so I have no code lying around related to that. > > > > The best that I know of is the work by anarcat and others, maintained at > > https://ikiwiki.info/tips/convert_moinmoin_to_ikiwiki/ > > > > I have some unfinished code converting a relatively large ikiwiki site > > to Zola, which might have some use also for migrating to other systems > > like hugo. > > When this migration happened back then, how far did it get? Stuck at 90% > or much earlier? If by "this" you mean the last of the 3 you quote above, then it is a migration not from MoinMoin but from ikiwiki to zola - I guess it is 90% done (but the remaining 10% may be the hardest to tackle). For the other earlier migrations, I don't know. Perhaps ask Frank Lin and anarcat. - Jonas -- * Jonas Smedegaard - idealist & Internet-arkitekt * Tlf.: +45 40843136 Website: http://dr.jones.dk/ * Sponsorship: https://ko-fi.com/drjones [x] quote me freely [ ] ask before reusing [ ] keep private -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 931 bytes Desc: signature URL: