Category Archives: technical things

Robust child process management in NodeJS?

Robust child process management in NodeJS? Not so easy :-).

For example, calling child_process.spawn() with a bad path does not throw an error right away. Although that error (ENOENT in this case, on Linux) is indeed known to the runtime in a synchronous fashion (NodeJS uses uv_spawn() for invoking the new process and that reports errors synchronously to its caller).

Even when child_process.spawn() fails to create a process it will return a process handle. Which you can try to kill(). And that would not throw an error right away.

To discuss this topic I submitted https://github.com/nodejs/node/issues/30668 a while back (as of which we improved the documentation for kill()) and https://github.com/nodejs/node/issues/33566.

Some more aspects and links, in no particular order:

A successful (but I think a little too involved) recipe for handling startup errors might be this:

  1. spawn()
  2. attach error handler, and upon error store the error state in a magic variable.
  3. implement and start a “success criterion watcher” (may even be based on polling) and in that logic consider the magic variable (for example, if it is known to be an error object, throw it).

 

 

 

VS Code: I cannot quite figure out how to search and replace in a selection

Gosh, search and replace in a selection of text in VS Code drives me nuts.

I can’t even describe how exactly it does so, but it works against my intuition.

It’s funny how in 2019 we haven’t quite settled on UX for that!

For me, it’s super hard to grasp how this (search and replace in a selection of text) is supposed to work in VS Code, as in: how would the people who designed this part of the product like me to use it? Which workflow did they have in mind? I don’t seem to understand the magic behind it. I often seem to end up replacing things in the entire document as opposed to in the selection that I just carefully created a second before. All I can conclude so far is that trial and error don’t seem to get me far :-).

Trying to understand the behavior based on docs and reading GitHub issues then reveals that this thing is complex, that it can be configured with a number of special configuration options, and that many users experience WTF moments on a daily basis. Users keep posting videos and GIFs of their weird experiences (thanks!). Two of these WTF moments are well captured in the GIFs in this comment: https://github.com/microsoft/vscode/issues/17563#issuecomment-542656591 (from October 2019).

I can say that search and replace in a selection of text works much better, so much more intuitively, in Sublime Text 3 (by default).

In VS Code it turned out that setting editor.find.autoFindInSelection to true helps me quite a bit towards getting more predictable outcomes.

Feel free to leave a comment below if you have an anecdotal opinion about all this.

If you’d like to read along, I find these issues pretty entertaining:

I particularly agree with this statement from November 2019:

I appreciate all the work and attention that has gone into this, but I don’t think I will ever be truly comfortable with Find/Replace in VS Code. The issue for me is that the “find in selection” button looks and feels like a toggle switch, not a trigger (or “fire button”).

If you flip a toggle switch from one position to the other, and then back to the original position, it should be as though you had never flipped the switch at all. With this mental model, it is always a shock and a disappointment when flipping the switch irreversibly destroys my carefully crafted manual selection.

 

 

 

Bokeh: disable touch interaction (disable drag, zoom, pan)

Bokeh is quite cool. I was looking for a way to disable touch interaction with a plot, though. Was a little tedious to find a solution on the web. Found it, buried in the docs. Therefore this quick post.

Say you have created a figure object with

fig = bokeh.plotting.figure(...)

Then you can disable the individual touch controls with e.g.

fig.toolbar.active_drag = None
fig.toolbar.active_scroll = None
fig.toolbar.active_tap = None

Very easy to do, you just need someone to tell you :-).

(Tested with Bokeh 2.0.0)

Bulma: sticky footer (flexbox solution)

Bulma is nice. I was looking for a way to get a sticky footer, though. Like many others, I was a little surprised that it’s not built-in: https://github.com/jgthms/bulma/issues/47

It’s of course doable. I have created a demo / minimal working example based on the solution proposed here.

<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />
    <title>Footer at the bottom / sticky footer</title>
    <link
      rel="stylesheet"
      href="https://cdn.jsdelivr.net/npm/bulma@0.8.0/css/bulma.min.css"
    />
    <style type="text/css" media="screen">
      body {
        display: flex;
        min-height: 100vh;
        flex-direction: column;
      }
 
      #wrapper {
        flex: 1;
      }
    </style>
  </head>
  <body>
    <div id="wrapper">
      <section class="hero is-medium is-primary is-bold">
        <div class="hero-body">
          <div class="container">
            <h1 class="title">
              The footer is at the bottom. Seriously. 🦠
            </h1>
            <h2 class="subtitle">
              By <a href="https://gehrcke.de">Jan-Philip Gehrcke</a>
            </h2>
          </div>
        </div>
      </section>
    </div>
    <footer class="footer">
      <div class="content has-text-centered">
 
          I am down here.
 
      </div>
    </footer>
  </body>
</html>

I posted this in the GH thread, too: https://github.com/jgthms/bulma/issues/47#issuecomment-603547213

Covid-19: HTTP API for German case numbers

Landing page: https://covid19-germany.appspot.com

The Robert Koch-Institut is certainly a cool organization, but I doubt they understand the role of (HTTP) APIs for data exchange. I believe that government institutions still vastly underestimate the power of collaboration on data.

Who would have believed that during a pandemic in 2020 we communicate current numerical data such as case counts via PDF documents or complex websites that can only be scraped with brittle tooling and headless browsers?

I closely monitored the situation for days, asked people, asked organizations. Nothing.

Now I have buit an HTTP API, providing the currently confirmed case numbers of Covid-19 infections in Germany:

https://covid19-germany.appspot.com/now

The primary concerns are:

  • convenience (easy to consume for you in your tooling!)
  • interface stability
  • data credibility
  • availability
$ curl https://covid19-germany.appspot.com/now 2> /dev/null | jq
{
  "current_totals": {
    "cases": 9348,
    "deaths": 25,
    "recovered": 72,
    "tested": "unknown"
  },
  "meta": {
    "contact": "Dr. Jan-Philip Gehrcke, jgehrcke@googlemail.com",
    "source": "zeit.de (aggregates data from individual ministries of health in Germany)",
    "time_source_last_consulted_iso8601": "2020-03-18T00:11:24+00:00",
    "time_source_last_updated_iso8601": "2020-03-17T21:22:00+01:00"
  }
}

This is served by Google App Engine in Europe. The code can be found here: https://github.com/jgehrcke/covid-19-germany-gae

I plan to

  • add time series data
  • add more localized data for individual states (Bundesländer)
  • enhance caching

Feel free to use this. Feedback welcome.

Huge shoutout to zeit.de for doing the work of aggregating the numbers published by individual ministries of health.

For historical data, by all means and purposes as of today I recommend consuming https://github.com/CSSEGISandData/COVID-19. For getting the current state, use the zeit.de data exposed via the HTTP API described above.

For now, I am sure that the current case count as provided by zeit.de is the best in terms of credibility and freshness. The actual underlying data sources are all official: these are the individual ministries of health.

The individual ministries publish their numbers usually once or twice during different times of the day. The journalists from zeit.de try to incorporate these data points as quickly as possible right after publication, also during the afternoon and evening. In contrast to that, the Robert Koch-Institut (RKI) may incorporate a specific update from a specific health ministry only after 1-2 days.

The RKI also doesn’t do what I call an atomic sum, but instead seems to sum numbers published by different health ministries at vastly different times: the RKI tries to find one number per day, and that number is not found during the evening (after “all data has come in” from the individual states), but seemingly at some unfortunate mid-day point in time where some individual ministries of health have just delivered a fresh update for the day, and others didn’t yet. Non-atomic.

This explains why, for example, the RKI’s official number for March 17 was ~7000 confirmed cases, whereas zeit.de already reported ~9300 at the same time (biggest contributor here is specifically that the last update from Nordrhein-Westfalen from March 17 didn’t make it into RKI’s sum for March 17).

 

Update: an official statement of the RKI about the delays in data processing, in German:

In Deutschland übermitteln die rund 400 Gesundheitsämter mindestens einmal täglich (in der aktuellen Lage noch häufiger) pseudonymisierte Daten zu bestätigten COVID-19-Fällen auf Grundlage des Infektionsschutzgesetzes elektronisch an die Bundesländer. Die wiederum übermitteln die Daten zu den COVID-19-Fällen elektronisch an das RKI. Für die Berichterstattung wird seit 18.03.2020 täglich der Datenstand 00:00 Uhr verwendet.

Zwischen dem Bekanntwerden eines Falls vor Ort, der Meldung an das Gesundheitsamt, der Eingabe der Daten in die Software, der Übermittlung an die zuständige Landesbehörde und von dort an das RKI liegt eine gewisse Zeitspanne. Die kann gemäß den Vorgaben im Infektionsschutzgesetz zwei bis drei Arbeitstage lang sein. In der aktuellen Lage erfolgt die Übermittlung deutlich schneller als im Routinebetrieb, weil Daten schneller verarbeitet werden. Dass einige Fälle mit etwas Verzögerung im Gesundheitsamt elektronisch erfasst werden, liegt auch daran, dass die Gesundheitsämter zunächst Ermittlungen zu den einzelnen Fällen und deren Kontaktpersonen durchführen und prioritär Infektionsschutzmaßnahmen ergreifen müssen, was die Ressourcen der Gesundheitsämter bereits stark in Anspruch nimmt. Ebenso werden die Daten am RKI validiert, um verlässliche Daten zu veröffentlichen. Auch innerhalb dieses Prozesses kann es zu geringen Verzögerungen kommen.

(source: https://www.rki.de/DE/Content/InfAZ/N/Neuartiges_Coronavirus/Fallzahlen.html)