December 1st, 2017

Anarcat: November 2017 report: LTS, standard disclosure, Monkeysphere in Python, flash fraud and Goodbye Drupal

Programing, Python, by admin.

Debian Long Term Support (LTS)

This is my monthly Debian LTS report. I didn’t do as much as I
wanted this month so a bunch of hours are reported to next month. I
got frustrated by two difficult packages: exiv2 and libreoffice.

Exiv

For Exiv2 I first reported the issues upstream as requested in
the original CVE assignment. Then I went to see if I could
reproduce the issue. Valgrind didn’t find anything, so I went on to
test the new ASAN instructions that tell us how to build for ASAN
in LTS. Turns out that
I couldn’t make that work either. Fortunately, Roberto was able
to build the package properly and confirmed the wheezy version as
non-vulnerable, so I marked the three CVEs as not-affected and moved
on.

Libreoffice

Next up was LibreOffice. I started backporting the patches to wheezy
which was a little difficult because any error in the backport takes
hours to find because LibreOffice is so big. The monster takes about
4 hours to build on my i3-6100U processor – I can’t imagine how long
that would take on a slower machine. Still, I managed to
get patches that mostly builds. I say mostly because while
most of the code builds, the tests fail to build. Not only do they
fail to build, but they even segfault the linker. At that point, I
had already spent too many hours working on this frustrating loop of “work/build-wait/crash” that I gave up.

I also worked on reproducing a supposed regression associated with the
last security update. Somehow, I couldn’t reproduce either – the
description of the regression was very limited and all suggested
approaches failed to trigger the problems described.

OptiPNG

Finally, a little candy: an easy backport of a simple 2-line patch for
a simple program, OptiPNG that, ironically, had a vulnerability
(CVE-2017-16938) in GIF parsing. I could do hundreds of
those breezy updates, they are fun and simple, and easy to test. This
resulted in the trivial DLA-1196-1.

Miscellaneous

LibreOffice stretched the limits of my development environment. I had
to figure out how to deal with out of space conditions in the build
tree (/build) something that is really not obvious in sbuild. I
ended up documenting that in a new troubleshooting section in the
wiki.

Other free software work

feed2exec

I pushed forward with the development of my programmable feed
reader, feed2exec. Last month I mentioned I released the 0.6.0
beta: since then 4 more releases were published, and we are now at the
0.10.0 beta. This added a bunch new features:

  • wayback plugin to save feed items to
    the Wayback Machine on archive.org
  • archive plugin to save feed items to the local filesystem
  • transmission plugin to save RSS Torrent feeds to
    the Transmission torrent client
  • vast expansion of the documentation, now hosted
    on ReadTheDocs. The design was detailed with a tour of the
    source code and detailed plugin writing instructions were added to
    the documentation, also shipped as a feed2exec-plugins
    manpage.
  • major cleanup and refactoring of the codebase, including standard
    locations for the configuration files, which moved

The documentation deserves special mention. If you compare
between version 0.6 and the latest version you can see 4 new
sections:

  • Plugins – extensive documentation on plugins use, the design
    of the plugin system and a full tutorial on how to write new
    plugins. the tutorial was written while writing the archive
    plugin, which was written as an example plugin just for that
    purpose and should be readable by novice programmers
  • Support – a handy guide on how to get technical support for
    the project, copied over from the Monkeysign project.
  • Code of conduct – was originally part of the contribution
    guide. the idea then was to force people to read the Code when they
    wanted to contribute, but it wasn’t a good idea. The contribution
    page was overloaded and critical parts were hidden down in the
    page, after what is essentially boilerplate text. Inversely, the
    Code was itself hidden in the contribution guide. Now it is
    clearly visible from the top and trolls will see this is an ethical
    community.
  • Contact – another idea from the Monkeysign project. became
    essential when the security contact was added (see below).

All those changes were backported in the ecdysis template
documentation and I hope to backport them back into my other projects
eventually. As part of my documentation work, I also drifted into the
Sphinx project itself and submitted a patch to make manpage
references clickable
as well.

I now use feed2exec to archive new posts on my website to the internet
archive, which means I have an ad-hoc offsite backup of all content I
post online. I think that’s pretty neat. I also leverage
the Linkchecker program to look for dead links in new articles
published on the site. This is possible thanks to a Ikiwiki-specific
filter to extract links to changed files from the Recent Changes RSS
feed.

I’m considering making the parse step of the program pluggable. This
is an idea I had in mind for a while, but it didn’t make sense until
recently. I described the project and someone said “oh that’s
like IFTTT“, a tool I wasn’t really aware of, which connects
various “feeds” (Twitter, Facebook, RSS) to each other, using
triggers. The key concept here is that feed2exec could be made to read
from Twitter or other feeds, like IFTTT and not just write to
them. This could allow users to bridge social networks by writing only
to a single one and broadcasting to the other ones.

Unfortunately, this means adding a lot of interface code and I do not
have a strong use case for this just yet. Furthermore, it may mean
switching from a “cron” design to a more interactive, interrupt-driven
design that would run continuously and wake up on certain triggers.

Maybe that could come in a 2.0 release. For now, I’ll see to it that
the current codebase is solid. I’ll release a 0.11 release candidate
shortly, which has seen a major refactoring since 0.10. I again
welcome beta testers and users to report their issues. I am happy to
report I got and fixed my first bug report on this project this
month.

Towards standard security disclosure guidelines

When reading the excellent State of Opensource Security report,
some metrics caught my eye:

  • 75% of vulnerabilities are not discovered by the maintainer

  • 79.5% of maintainers said that they had no public-facing disclosure
    policy in place

  • 21% of maintainers who do not have a public disclosure policy have
    been notified privately about a vulnerability

  • 73% of maintainers who do have a public disclosure policy have been
    notified privately about a vulnerability

In other words, having a public disclosure policy more than triples
your chances of being notified of a security vulnerability. I was also
surprised to find that 4 out of 5 projects do not have such a
page. Then I realized that none of my projects had such a page, so I
decided to fix that and fix my documentation templates (the
infamously named ecdysis project) to specifically include
a section on security issues.

I found that the HackerOne disclosure guidelines were pretty
good, except they require having a bounty for disclosure. I understand
it’s part of their business model, but I have no such money to give
out – in fact, I don’t even pay myself for the work of developing
the project, so I don’t quite see why I would pay for disclosures.

I also found that many projects include OpenPGP key fingerprints in
their contact information. I find that’s a little silly: project
documentation is no place to offer OpenPGP key discovery. If security
researchers cannot find and verify OpenPGP key fingerprints, I would
be worried about their capabilities. Adding a fingerprint or key
material is just bound to create outdated documentation when
maintainers rotate. Instead, I encourage people to use proper key
discovery mechanism like the Web of trust, WKD or
obviously TOFU which is basically what publishing a fingerprint
does anyways.

Git-Mediawiki

After being granted access to the Git-Mediawiki project last
month, I got to work. I fought hard with both Travis and Git, and
Perl, and MediaWiki, to add continuous integration in the
repository. It turns out the MediaWiki remote doesn’t support newer
versions of MediaWiki because the authentication system changed
radically. Even though there is supposed to be
backwards-compatibility, it doesn’t seem to really work in my cases,
which means that any test case that requires logging into the wiki
fails. This also required using an external test suite instead of the
one provided by Git, which insists on building Git before being used
at all. I ended up using the Sharness project and submitted
a few tests that were missing.

I also:

I also opened a discussion regarding the versioning scheme and
started tracking issues that would be part of the next release. I
encourage people to get involved in this project if they are
interested: the door is wide open for contributors, of course.

Rewriting Monkeysphere in Python?

After being told one too many times that Monkeysphere doesn’t support
elliptic curve algorithms, I finally documented the issue and
looked into solutions. It turns out that a key part of Monkeysphere is
a Perl program written using a library that doesn’t support ECC
itself, which makes this problem very hard to solve. So I looked into
the PGPy project to see if it could be useful and it turns out
that ECC support already exists – the only problem is the specific
curve GnuPG uses, ed25519, was missing. The
author fixed that but the fix requires a development version of
OpenSSL, because even the 1.1 release doesn’t support that curve.

I nevertheless went ahead to see how hard it would be to re-implement
that key component of Monkeysphere in Python, and ended up
with monkeypy, an analysis of the Monkeysphere design and first
prototype of a Python-based OpenPGP / SSH conversion tool. This lead
to a flurry of issues on the PGPy project
to fix UTF-8 support, easier revocation checks and a few
more. I also reviewed 9 pull requests that were pending in the
repository. A key piece missing is keyserver interaction and I made
a first prototype of this as well.

It was interesting to review Monkeysphere’s design. I hope to write
more about this in the future, especially about how it could be
improved, and find the time to do this re-implementation which could
mean wider adoption of this excellent project.

Goodbye, Drupal

I finally abandoned all my Drupal projects, except the
Aegir-related ones, which I somewhat still feel attached to, even
though I do not do any Drupal development at all anymore. I do not use
Drupal anywhere (unless as a visitor on some websites maybe?), I do
not manage Drupal or Aegir sites anymore, and in fact, I have
completely lost interested in writing anything remotely close to PHP.

So it was time to go. I have been involved with Drupal for a long
time: I registered on Drupal.org in March 2001, almost 17 years
ago. I was active for about 12 years on the site: I made
my first post in 2003 which showed I was already “hacking core”
which back then was the way to go. My first commit, to the
mailman_api project, was also in 2003, and I kept contributing until
my last commit in 2015, on the Aegir project of course.

That is probably the longest time I have spent on any software
project, with the notable exception of the Debian project. I had a lot
of fun working with Drupal: I met amazing people, traveled all over
the world and made powerful websites
that shook the world. Unfortunately, the Drupal project has
decided to go in a direction I cannot support: Drupal 8 is a monster
that is beyond anything needed for me, my friends or the organizations
I support. It may be very well for the startups and corporations, to
create large sites, but it seems it completely lost touch with its
roots. Drupal used to be a simple tool for “community plumbing
(ref), it is now a behemoth to “Launch, manage, and scale
ambitious digital experiences—with the flexibility to build great
websites or push beyond the browser
” (ref).

When I heard a friend had his small artist website made in Drupal, my
first questions were “which version?” and “who takes care of the
upgrades?” Those questions still don’t seem to be resolved in the
community: a whole part of the project
is still stuck in the old Drupal 7 world, years after the first
D8 alpha releases. Drupal 8 doesn’t address the fundamental issue of
upgradability and cost: Drupal sites are bigger than they ever
were. The Drupal community seems to be following
a very different way of thinking (notice the Amazon Echo device
there) than what I hope the future of the Internet will be.

Goodbye, Drupal – it’s been a great decade, but I’m already gone and
I’m not missing you at all.

Fixing fraudulent memory cards

I finally got around testing the interesting f3 project with a
real use case, which allowed me
to update the stressant documentation
with actual real-world results. A friend told me the memory card
he put in his phone had trouble with some pictures he was taking with
the phone: they would show up as “gray” when he would open them, yet
the thumbnail looked good.

It turns out his memory card wasn’t really 32GB as advertised, but
really 16GB. Any write sent to the card above that 16GB barrier was
just discarded and reads were returned as garbage. But thanks to the
f3 project it was possible to fix that card so he could use it
normally.

In passing, I worked on the f3 project
to merge the website with the main documentation, again using the
Sphinx project to
generate more complete documentation. The old website is
still up, but hopefully a redirection will be installed soon.

Miscellaneous

  • quick linkchecker patch, still haven’t found the time to make
    a real release

  • tested the goaccess program to inspect web server logs and get
    some visibility on the traffic to this blog. found two limitations:
    some graphs look weird and would benefit from
    a logarithmic scale and it would be very useful to
    have plain text reports. otherwise this makes for a nice
    replacement for the aging analog program I am currently using,
    especially as a sysadmin quick diagnostic tool. for now I
    use this recipe to send a HTML report by email every week

  • found out about the mpd/subsonic bridge and sent a few issues
    about problems I found. turns out the project is abandoned and this
    prompted the author to archive the repository. too bad…

  • uploaded a new version of etckeeper into Debian, and started
    an upstream discussion about what to do with the pile of
    patches waiting there

  • uploaded a new version of the also aging smokeping in
    Debian as well to fix a
    silly RC bug about the rename dependency

  • GitHub also says I “created 7 repositories”, “54 commits in 8
    repositories”, “24 pull requests in 11 repositories”, “reviewed 19
    pull requests in 9 repositories”, and “opened 23 issues in 14
    repositories”.

  • blocked a bunch of dumb SMTP bots attacking my server
    using this technique. this gets rid of those messages I was
    getting many times per second in the worst times:

    lost connection after AUTH from unknown[180.121.131.155]
    
  • happily noticed that fail2ban gained the possibility
    of incremental ban times in the upcoming version 0.11, which
    i documented on Serverfault.com. unfortunately, the Debian
    package is out of date for now…

  • tested Firefox Quantum with great success: all my extensions are
    supported and it is much, much faster than it was. i do miss the
    tab list drop-down which somehow disappeared and I also miss a
    Debian package. I am also concerned about the maintainability of
    Rust in Debian, but we’ll see how that goes.

I spent only 30% of this month on paid work.

Back Top

Leave a Reply