Dmitry Petrov

From the basics to complexity and back

2022-10-26T00:00:00+02:00

Intro

Do you remember good old nineties? It was a time when websites looked, well, like a mess on today’s standards and browsers were not really helping with that. You could make a career on being able to craft round corners that looked ok on netscape, opera and internet explorer, becase it wasn’t trivial at all.

What happened next was a huge movement that gave us all the web standards we love and use today. It was also the time of pixel perfect design. I still remember how I would alt-tab between the browser window and photoshop or use an exported png as a background in order to fulfill designer’s expectations.

Not only that, we were all busy crafting our home pages, making them look unique and awesome. Or take MySpace or Livejournal.com where you could make your journal unique by adjusting the templates.

That was an era full of creativity but then it was gone and what’s interesting many people haven’t noticed.

Facebook

Disclaimer - I was a livejournal.com engineer back in the days. It’s hard to overestimate how much users cared about their styles. Every time a change had to be introduced (e.g. fb like button) we had to carefully craft it in all the different designs that were supported by the system, help out popular blogs that had their own custom styles and still got it wrong once in a while.

The same users who hated us for any changes on their beloved blogs and eventually switched to facebook and other huge platforms that are there now didn’t care a single bit about the fact that they had near zero level of control on the presentation there. As it turned out, if the content is good and all your friends are around you don’t need that much. And facebook is actually not very extreme in this situation, you can craft your post in markdown while other platforms don’t give users almost any way to control the layout and everyone is just fine with it.

Instagram users can only write plaintext only without any formatting and it even doesn’t look nice on the phone and you have to insert an image even in case you only want to post a message and you know what? Nobody cares. I can only talk about russian speaking segment of the platform but the amount of activity there is just crazy. People not only share cat photos there, but also buy and sell stuff, write analytical articles, do workshops and trainings without any problem. The fact that from time to time they have to literally edit the photo and put a text overlay on top of it (and it often looks like some of the worst myspace examples) doesn’t bother anyone. And all the communication is solved with a restricted instagram account or a closed whatsapp group.

Or take twitter. Before we got threads some users posted images with the text to work around the text limitation. That’s crazy, you cannot even copy/paste or quote this thing. And no one cared. As it turns out, the main reason people are on the internet is social interactions and if the design is good enough and, most importantly, others can discover your content, the rest simply does not matter that much.

Do you remember how your friend’s website looks like? You don’t probably, because it actually doesn’t matter that much and what even more probable they don’t even have one. But you probably do remember some catchy text they wrote. I’m a huge fan of Dan Luu’s blog and eventhough it’s literally just html in it’s simplest form, just try to open one of his posts in the reader mode in firefox and you’ll find out that it looks awesome with zero bloat you usually get. It doesn’t even use react!

Same goes for your favorite rss reader where all articles look nice even though the author doesn’t have to do anything with it. You can go on github and open a file with this very post and it will render just fine for you, zero html or styling involved from my side.

HTML

Now let’s discuss what unites social platforms, rss readers and firefox reader mode.

If you compare all of them you will see that they all do not support any of the complex markup of html or formatting of the css. In most cases users cannot even chose where to put the images, it’s all decided by the platform.

If you think about it all the platforms have become real browsers within a browser and what makes them so effective is specifically that the users are limited in the way the present content. Why so? Because simple markup makes it trivial to render the content anywhere ranging from a desktop browser to a fridge. Not just that, it’s absolutely up to platforms how to render the content in the first place. The photos are at the bottom now, but they can as well be put anywhere else, page previews can be generated and a whole lot of other stuff that would be simply not possible if the constraints were not there. It’s also important to note again that users usually don’t care too much about the presentation of the information on the platform if it reaches their audience, because this is the only point.

If you still have any doubts, take Gemini proctocol as an example. The list of clients on the project page suggests that eventhough building a proper web browser is no small task, the rendering part of it is definitely trivial especially if you compare it to let’s say a multiyear effort by Serenity OS community to get ther web browser to a proper level. They’re doing wonders really, but the complexity and scale of specs and the variety of real world use cases requires a tremendeous amount of work to get it right to an extent that even bigger players like Microsoft and Opera have given up on supporting their own rendering engines.

Another angle to take is that there is another problem that’s being discussed every now and then - as time goes by, the web is not getting any faster. We get better connection only to get more bloated websites as a result. A basic website can deliver megabytes and megabytes of styles and scripts just because an unlucky dependency made webpack include a half of npm registry into the build. You can fight against it, but that’s really an uphill battle, everything is designed in a way to make including more stuff simpler, not the opposite. The CSS design is fundamentally hostile to any cleanups, which most people who tried to remove a div p a selector from the code base would agree with.

Just to make the final point: none of the dialects used by the platforms is standardised. Markdown has a standard, but except from really basic stuff that covers suprisingly big amount of cases, the rest is not really well supported everywhere and again that doesn’t bother too many people.

So what?

Now that I’ve built up this argument, the question is also there - what can we do with this observation?

First and foremost we can start by acknowledging the fact that html, css and js are gone as creativity tool for most of people - too much complexity and bloat.

Second, suprisingly small amount of features is needed for an ordinary person to start producing amazing work. We’ve already been there with Macromedia Flash. If somebody still remembers Action script 1/2, it was a joke not only because of the lacking features, but also because most of it was often embedded into the frames in a very chaotic manner by people who barely knew how to code. The most surprising fact is it still looked and worked amazingly. Every topic on hacker news related to flash instantly gathers a lot of comments from users longing for that content.

The beauty of it was not only that there were tons of it, but also that it was really amateur, anyone could do it and my stong belief lies in fact that no big corp or data science can beat that. I think that’s what we’re missing now.

You may also not admit it but you probably also think about good old days when creating a website was as easy as copying an index.php file to a remote ftp.

We can leave web technologies for big guys who want to make amazon.com or google docs, however we certainly need something else to unleash a new era of creativity on the web that also doesn’t mean getting into another walled garden.

My bet would be on the fat clients. They are pretty tough to evolve sometimes, just remember how long we’ve been waiting for the official death of IE6, but at the same time that’s the place where the real changes can happen and rss, browser reader mode, gemini and flash proved it. My bet is that the new thing should be simple again, yet complete enough to allow artistic expression.

Should it be one more flash generation or something else? I don’t know, but what’s certain is that it doesn’t need to have that much to become the next big thing. It will get big and standardized and bloated in time but before that we will have some very fun years.

Thanks a lot to Artur Burtsev and Artem Tyurin for the comments and ideas.

Mcdonald's and web servers

2022-10-04T00:00:00+02:00

When you have a hammer, everything looks like a nail they say. That’s true to an extent, but actually some patterns are so generic that both problems and solutions seem to be the same. If you think otherwise just let me to tell you a story of one fast food restaurant.

It’s quite rare to find an inefficiently running place in the Netherlands and almost impossible to get into a real mess and the whole restaurant part of mcdonald’s business is based on the efficiency of the operations. Given this I was quite surprised to get into a restaurant that literally almost collapsed under the load. It’s worth saying that currently 99% of the orders are being made via the terminals near the entrance.

When I entered the place I has not been initially alarmed by a huge crowd near the pick up area and I made my order only to realize that not only my order did not show up on the screen but it looked like there were 13 orders before mine waiting for their place on the screen.

The kitchen seemed to be in a bad situation:

There was a pile of hamburgers of all sorts ready for collection
A guy responsible for drinks kept doing his part and a tray in front of him had drinks for something like 30 orders.
Orders were served in a very slow pace.

Given such a strange mixture I started inspecting the area in hope to find the source if the problem. Soon it became clear that the root cause was a kids party! What happened was that parents ordered something like 30 Happy meals and fulfilling this order cause such a degradation in the throughput that the restaurant was not able to keep up with new orders anymore.

What made the problem worse was that since kitchen kept preparing orders that it has not been able to serve in time, all these poor hamburgers caused a lot of contention and penalized the throughput even more. Not only this, since a lot of food got cold before it was served and customers waited for way too long, the restaurant had to deal with chargebacks as well as replacements where some of the food had to prepared again which meant a lot of waste being generated instead of a useful output. And since the situation was quite extraordinary, all the workers were in the kitchen with no staff to clean waste bins and after they got overflowed it of course took considerably more time to clean them.

Apparently the workers were doing their job but the workers were doing their job but the system itself was out of tune. If you think of it, the problem could have been mitigated early:

If we’re not talking about ad hoc decision, the chain could have trained workers to move quicker to achieve vertical scaling
Some of the workers could have moved to serving the requests instead of completing them in order to increase the throughput
The problem could have been made less severe if the restaurant simply did not allow to do such gigantic orders. If they were split into a set of smaller ones the load could have been distributed more evenly.
Alternatively, they could have put rate limiting in place by disabling some of the ordering terminals.

What’s the morale there? If you do a web service on a bigger scale, make sure that the rate-limiting is in place, requests are time-scoped and monitoring is in place to let you know in advance that your service needs some optimization or scaling. Have fun!

On bronze age collapse

2022-04-24T00:00:00+02:00

As it often happens, many patterns repeat themselves in many different areas, maybe in slightly different forms and shapes. You can have a look everywhere.

One of the theories behind the Late Bronze Age Collapse is that the civilization at time came into some sort of perfect storm - natural disasters, hunger caused wars that broken trade links between the countries and made it impossible to source all the components needed to made bronze and the civilization fell apart with a frigthening speed. Fast forward to late Rome scenario - historians still argue about the reasons of West Rome demise and I’m by no means an expert in the area but one theories that sounded plausible to me was that the efficient efficient trade was one of the main reasons that elevated the Rome

What do you read?

2020-04-24T00:00:00+02:00

There are some habits that I developed during my life that sometimes give me pure moments of joy. The reason that happens is mostly due to the fact that I don’t know exactly what I get and I don’t do it to achieve something, it’s plain curiosity.

One of them is browsing maps. Earlier that were paper maps. now it’s Google Maps but the principle stays the same - I just scroll through the map in any direction and sometimes find some interesting stuff (like this circular house in Moscow).

Another one is reading signs or just looking around at cars and buildings. Here is one of my recent findings. How cool is that?

Or here is one more - whenever I see someone reading a book I immediately start googling the name and in many cases I end up reading something fascinating and new to me.

What unites all of this is pure randomness of the process - you don’t know what you get. It can be total crap, it can be a pearl, you don’t know and this is the essence.

Anyone can name lot’s of random events that turned life upside down and that was the best thing happened to them. Moreover, in many cases it’s completely fair to say that random events played a far more important role that any specific plans.

What also unites all this pure randomness is that it’s impossible to optimize for, conversion is too low. Amazon or Spotify will be able to give you most relevant similar items to what you already have, but noone will be able to recommend you something that you don’t know you like, it’s always a bubble of some sorts.

It’s completely fine to be in such bubbles, since the experience is predictable and enjoyable, but you can already feel that you’re missing out and you don’t know what. Every single time randomeness disappears in one or the other area we gain something but at the same time we lose.

The closest to my heart is of course books. Since it’s all the smarphones and e-readers now, you cannot really say what people read, and that’s tragic, because some of them can read a book that will change one’s life. And no one will ask.

What I think would be really cool is to reverse this trend. Let every e-reader or smartphone to publish information via bluetooth to nearby devices. There is no point in that except feeding curiosity of others, but that already sounds fascinating, right? To do something for no reason, except letting others to experience a pure joy of randomness.

On Web without HTML

2020-04-07T00:00:00+02:00

Intro

Note: I use markdown throughout the text but you can replace it with you favorite comparable markup language of choice.

Internet is a very interesting place now. We have moved from a totally decentralized place to the one with very few giants in all fields (Amazon, Facebook, Instagram, Reddit, Medium). There is a lot of discussion about this state of affairs and I don’t want to discuss the obvious.

What I want to discuss is another wave, that came our way without too much publicity - simple markup languages like markdown. One might wonder - what’s so interesting about it? I think this wave is not accidental and goes along with big social platforms and decline of html/css hype.

Let’s start from the last point. Anyone who has been in web development comunity for a while remembers good old days when the movement for web standards was strong and the community itself was very vibrant. Lots of new hacks, tricks and technologies, a lot of discussions. I might very well be out of the bubble now, but from my observations it’s all gone now. People were obsessed with pixel perfect design back then, just remember all possible ways to get perfect round corners or opacity or any other design element. Where is it now? It’s still around us of course, but in many cases it lost it’s relevance. And the reason is not only because we got flexbox and new features in css. Most notably, content and community is the king, especially when visuals are good enough.

I still have my active account on Livejournal.com where people were people were obsessed with keeping their diary in their style and were very vocal about it. It was mainstream back then, but it isn’t anymore. It appeared that once you have people around, design becomes much less relevant. Instagram, Twitter and Facebook are good examples there. Instagram is extreme in this case - only plain text is allowed, no customizations at all. On twitter users used to post text as pictures to work around text limit. On Facebook there’s a bit more freedom in sence that you can use markdown but that’s it. None of them allow to do any significant customizations and all of them have lots of users that post valuable content, connect to the loved ones or do business there. With zero css and not caring about pixel perfect design a single moment. If you think about it, all these platforms became browsers in browser and all the markup and styles are out of play there. The reason why it’s so is because this form of content is much easier to handle. There are no special cases, everything is under platform control. It’s much easier to do a mobile version, it’s much easier to do an app on any type of hardware/software.

Next, let’s have a look on Markdown. It’s not about it in isolation, of course. It’s rise is only partially thanks to it’s simplicity, a lot of it is due to platforms like Facebook, Github or Gitlab, that render it in a good way along with links, which means you can have a site with hyperlinks for no cost and it will be accessible to users. As you see, it’s the same here, no css involved, all is needed is a browser in browser, that converts the markup to some tolerable form.

And we also have a huge movement of static websites. HTML and CSS are still there, but underneath it’s often the same markdown. We do design only once (like on this particular blog) and then work only with markdown, because it’s easy and good enough and what happens underneath is a conversion of markdown to a good enough html/css, I’ll put it into the same bucket with github/gitlab.

And to make things even more interesting - all major web browsers have a reader mode now that takes any wonderful well designed page with ads, noise etc takes the main content and renders it into a simple easy to read form. From what’s supported one might argue that an opposite conversion takes place - from HTML/CSS back to markdown with it’s limited support for headers, lists images and a bit more. Or you can take any of Dan Luu’s posts which look bad only till you enable the reader modex in firefox which makes them near perfect.

We didn’t talk about money so far. The situation improved quite a lot from early days in sense that there are solutions like Stripe now that allow to implement payment solutions with much less effort and there is Patreon that provides a distinct monetization strategy by allowing to support a person on ongoing basis. There are even projects like Substack.com, that allow to do paid newsletters. And we thought newsletters were dead.

The last part is community and there is a split there too. While back in the days everybody was busy integrating something like Disqus comments (this blog included), I have yet to see lively discussions happening in standalone blogs. Discussions happens in places where people actually hang out, hence they’re all on Hacker News, Reddit or big social networks now. Of course it’s possible to name some exceptions, but it’s safe to agree that it’s exactly that - an exception.

What that means is that you don’t need HTML/CSS to be popular and facilitate discussions. Two good exteme examples of this are blog of Dan Luu and essays of Paul Graham.

One can look at this trend from a different angle too. Semantic markup, precise styling, strict protocols were all about effectiveness. How do we describe page in the best way, that it will be technically possible to render it like we want. How do we integrate payments in our platform in the best possible way? It’s still there, but what is also there now is that people find new ways of working with restricted platforms that do not support a lot/any of it.

I can definitely say this for the russian part of the Instagram, but I guess it should be the same for other segments too. Users sell things, facilitate courses or workshops and handle a lot of things manually or by combining existing tools. Want to make a payment? Just transfer money on someone’s bank account. No need to make a separate courseware - yet another closed account or Whatsapp group is enough. Everything that software engineers hoped to solve in a perfect way was successfully hacked around by ordinary people with much more limited technical skills.

The last thing that’s worth talking about in this long intro is a concept of ownership and distribution. On classical websites author controled all of it. And you can see copyright notes all over the place. Does it really work on big social networks? Distribution is lost for sure. Once you make a post, it’s visibility is certainly out of your control. It’s up to facebook to put it in front of millions of users or never show it to anyone and the only tracking you can get is the one that’s provided by the platform and you only can trust it, the same goes for any other platform and there is nothing users can do about it. And what’s interesting people are fine with it too, even with reposts, since there is little left of the concept of a “place” of a person on the platform.

To sumarize, what people really want is:

A decent (not perfect) way to render content with some simple formatting
Discovery
Community
Some way to do payments that does not necesserily have to be integrated into a platform

And as you can see, it appeared that there is a very limited need in complex markup language, styling and scripting in this usecase. Moreover it can now be considered as a burden, since there is a lot to learn there, a lot of platforms to support, a hosting and related costs to worry about with modest benefits compared to social networks solution.

No HTML

Now, having said all of this, it’s time to think about if we still need html now in the form as it is now. For complext usecases we certainly do - it’s hard to implement amazon in markdown without extending it the extend that it becomes similar to HTML, but for many other cases it might be a really good fit. And the fact that it’s constrained is more of an advantage than a drawback.

For example, it’s much better for the bandwidth, can be easily stored offline. What’s more we move from pixel perfect design which requires effort from authors to browsers that are now responsible for all visual aspects.

Can we imagine a situation where we skip all the HTML/JS/CSS and serve a page in a language like markdown directly? It’s like going to pre-CSS era, but it doesn’t sound aweful anymore.

Also, it’s kind of solved on big social platforms, however can we take this bit out of them and put it in the wild?

Features we would like to preserve are identity, distribution and payments. None of it necessarily requires any complex markup to be solved, especially if we assume that browser takes care of it. So, what if we return to a concept of browser fully owning the presentation layer?

We can also split different aspects apart. It’s not required for all these features to be on one platform in case page itself has some metadata that can be used by the browser, and that also opens up an opportunity to build the best possible experience on this particular platform.

For example, if you want to sell the access to an article, you can encrypt it with your favorite payment provider and mention required data in the header, so that browser can display all necessary information to the user, facilitate a payment using open protocol with this provider and get a key to decipher it.

In the same way it’s possible to have a link to a site that facilitates discussions, however it’s up to browser to check your favorite one to see if a post is being discussed there.

If you think about, taking monetization strategy and discussion apart also means that you don’t really care if a page has been really served from your website, since there is no dependency on pageviews anymore.

All this does not mean that it has to be self hosted, but it might be coupled with rss to make it discoverable and different content aggregators can use it to show you personalized suggestions.

Moving all the rendering functionality to the browser also means that it becomes a point of extension. If you want to support a different type of content, you might need to have a different render, which might even be the same html/css, but it will be only one of possible options.

There is still place for them of course, especially for big application that will never fit into a constrained solution, and making a separate extension for all of them is probably not an option.

Complexity tower and a holy grail of web development

2020-01-03T00:00:00+01:00

I’ve been thinking about it for a while. Sometimes I was so bad that I wanted to quit programming, sometimes it got better and I though it makes sense again (sort of). But in reality, it’s bad, it’s really really bad. After watching Jonathan’s Blow video “Preventing the collapse of Civilization” I decided to finally convert some words into bytes. What am I talking about?

Let’s just observe how a perception of complexity kept changing with time in regards to different concepts of web development.

If you remember good old days, it was a pretty easy thing to make a website. You edit an index.html page till it looks fine, then you copy it to a cheap hosting with any tool you like and you’re done. If you really wanted to have some interactivity you could go wild and write some perl or PHP.

Maybe you also remember how local development worked back in the days. My weapon of choice back then was WAMP or something similar.

And that was it, mostly. There were several not so easy things with this approach. For example, you had to manually fiddle with the database, and CSS + HTML was a special kind of art due to amazing number of bugs and inconsistencies across different platforms.

Then the pendulum started it’s swing.

A couple of great things that happened back then was appearance of jQuery and other similar libraries like mootools and standards movement together with bulletproof css frameworks like Twitter Bootstrap that allowed an average developer to abstract away from the browser internals and just code a website.

On the other side of the stack framworks like Zend, Django or Rails turned backend development on it’s head and raised productivity many times, because data management, templates and routig became dead simple.

It was cool and nice, however with the rise of complexity on the frontend jQuery was not enough anymore. It was dead simple to do stuf like $('#block').slideUp() or even $('#popup').show(), but if you wanted to wire up something really complex, all that rather quickly turned into a spaghetti. Just remember all that script.js files that contained everything as one wall of text. The same goes for template engines - it was quite simple to split them into a couple of files, but a big number of templates was troublesome - most of them were text based (just remember that time when you spent hours hunting a missing closing tag), and scope was a real problem: with many layers it was harder to say where you got this specific variable from and when it’ll disappear.

Web development was booming and everyday someone popped up with a new idea on how to do it in a simple way and the whole web-development ecosystem turned into a “Hey, check out this cool hack” kind of community.

What happened next was containers, rise of single page applications, dynamic data formats like json or yaml (only old guys love xml, yes) and proper package management.

They made it so simple to share and compose, that you could barely resist. Now you could get even more efficient with what you do, especially with libraries and frameworks like React or Angular or, later Vue.

The only catch was that the day when you could just save a file and upload it to a cheap hosting was gone. Build step became a necessity and as things progressed even more, React and friends became a sort of default solution for any kind of application. You could easily do time machine kind of tricks, because your state was not managed with Redux, but a simple animation was not so simple anymore and having a popup was not something you would really like to do.

Moreover since your frontend turned into a full js app, serving an html page became something that you really need to think about. In the end a popular solution was to run a node.js app alongside with a backend app and consume backend apis both from the node.js app and from the frontend, we got isomorphic apps.

It was really cool, the only thing that you now had two runtimes instead of one, had to monitor and understand them and their interaction. The pain was not immediate, especially if you run them locally, but docker and docker compose made it so dead simple to run a lot of stuff in an easy manner, that you could hardly understand how such setups worked and you had to in the end because out of many compomnents the probability was quite high to have at least on of them brokend at least in one of the many environments.

And then we got kubernetes that made it quite simple to run such complex apps, but it was a beast by itself and now you had a magical thing running and it was wired up by a pile of yaml files that looked so easy to write but were fantastically hard to reason about when you had a lot of them.

Now you could do a lot of stuff. The only thing that you had trouble with was to get a development environment and speed like it was with WAMP stack.

One more side effect that we got was that backend frameworks somehow lost a lot of ground. Since everything is an spa, you can’t simply use Django forms now, you’ll have to leave that alone and do your form with react, which is not as simple. And don’t forget to do validations! And all this had to be joined with an api that you now had to write instead of just writing post handler there.

The other moment is a cloud push. It works amazingly well and allows you to run the craziest setup if you manage to wire it up but what it makes challenging is having an idea about your bill next month.

There was so much frustration at this point that it’s quite reasonable that the whole frontend world embraced Typescript to get a bit more sanity in everything happening.

And btw, I’m not even talking about all the regulations that came our way - think about all the EU regulations that made it kind of safer for us, except that everybody agrees on anything anyway just to get thought that pile of cookie related popups to read an article. One can say that we’re in a better place as customers now, but I think that all big guys will implement/find a workaound to it anyway, so in reality we just drastically increased the load on all small companies that have to deal with it. When did you ask a random website to delete all your data last time?

Now, I think it’s time to make a step back and think for a bit.

Everything I mentioned so far was really necessary for some projects. I mean it - docker is really powerful, you can do wonders with new js framworks, however what’s not there is the simplicity of doing things in a way that was achievable.

And while it’s really nice that all new stuff takes literally seconds to start, it’s not what we’re doing every day. What we’re doing is maintenance and iterations on a codebase. If your setup requires 200mb of javascript and some more python to run with some yaml on top, it’s not simple to maintain and quite frustrating to deploy. No wonder developers go for solutions like Heroku just to relieve some pain.

And while all big companies require these tools we should not. If what you’re building is a not so big website, you should be able to do it in a simple to maintain and deploy manner with as little moving parts as possible, while still being able to take an advantage of last years of web development.

It’s time to think again and come up with and promote a solution that will be dead simple in implementation and still allow to evolve to a complex one if necessary but not before.

Part 5: Gems and pitfalls

2018-06-30T00:00:00+02:00

All the points below summarize the things I enjoy using in Common Lisp and are not especially the ones preferred or recommended by community. Also proof reading showed that many of the examples were already mentioned earlier, but I still think it’s good to have them all in one place for the reference. Here you go.

Generic functions

If I were to choose the single most amazing feature of the language I would choose generic functions. Why?

The reason is that decoupling them from the object resulted in amazing freedom and flexibility when using them.

First of all, generic function dispatch against all it’s arguments and second, all of the arguments can be of any type and not some object hierarchy. What the means that it’s very easy to build recursive implementations that do some preliminary work with arguments and then call the same function again which makes generic function dispatch differently and execute another implementation. I used this trick a lot.

One example of such a definition could be fetch-posts function. What we know that we want to fetch posts to some kind of database, which has a store. Such a definition means that we can have objects of both types at our disposal. With described technique it’s simple to make it work as desired:

(defgeneric fetch-posts (db))

(defmethod fetch-posts ((db <db>))
  (set-credentials db)
  (fetch-posts (fetch-store db)))

(defmethod fetch-posts ((store <store>))
  (do-actual-fetch))

When we call fetch-posts against the database implementation just extracts store from it and call the function again which execute another method and now we can fetch posts both with store or database instance without any effort.

Another bonus from decoupling is that you are free to create any new function and implement it against any object hierarchy. It may be not even a hierarchy but just a set of types.

Remember function to-hash-table that I used before? It is implemented as a generic function.

(defgeneric to-hash-table (source &key))

With such a definition you can just implement it against any type that you want without any restrictions. When I needed it for the database I wrote this implementation:

(defmethod to-hash-table ((db <db>) &key (key-sub #'itemid))
  (let ((ht (make-hash-table :test 'equal)))
    (dolist (post (posts db) ht)
        (setf (gethash (funcall key-sub post) ht) post))))

And when later it became clear that <store> class can also benefit from one, I just implemented it:

(defmethod to-hash-table ((store <store>) &key)
  (let ((ht (make-hash-table :test 'equal)))
    (dolist (item (events store) ht)
      (let ((itemid (-<> item
                         (getf <> :event)
                         (getf <> :itemid)))
            (ts (getf item :sync-ts)))
        (setf (gethash itemid ht) ts)))))

Having said that I need to mention that this freedom applies to external functions as well and many of the external libraries expose precisely generic functions to control their behavior. The good example is with plump library when it was enough just to implement function against the set of types without doing any work at all with types.

Last bit that I want to praise is auxiliary methods. This is just incredible because of flexibility it gives. When I just started coding I wanted to have database saved on any operation. Post created? Save! Updated? Save! Deleted? Save!

Using :after modified allowed me to completely decouple the logic and for example, the main implementation of publish-post knew nothing about saving to the database, but meanwhile, in the other package it was as easy as

(defmethod publish-post :after ((db <db>) (post-file <post-file>))
  (save-posts))

And it’s done.

Another use case can be found in markdown handling that I did. What I wanted to do was to write a file with links pointing to the local files and translate them to the real urls on the markdown compilation phase.

(defmethod render-span-to-html :before
    ((code (eql 'inline-link)) body encoding-method)
  (let ((record (cl-journal.db:get-by-fname cl-journal::*posts* (cadr body))))
    (if record
        (setf (cadr body) (cl-journal.db:url record)))))

:before method does not in general change the behavior of the main code, however, it has access to all it’s arguments and in this case I check if whatever is passed as an url can be resolved to a post url from the database and modify argument accordingly. After that, I don’t really need to touch library logic, it works as usual.

Streams

Streams are really powerful. Languages that don’t have them (let’s say javascript) often end up with ugly hacks or additional code or with string concatenation all over the place and with streams all the api gets streamlined immediately and the neat thing is that it’s still totally under control.

I benefited from this fact quite a lot. One example is saving data to the file. Common Lisp provides a function pprint that prints data structure in a nice way. Default is stdout, but stream can also be passed as an argument. Given that saving state to file becomes as simple as:

(defun save-posts ()
  (with-open-file (out *posts-file*
                       :direction :output
                       :if-exists :supersede)
    (with-standard-io-syntax
      (pprint (to-list *posts*) out))))

And what’s cool is that whatever is printed with pprint by the definition can be consumed by read. Since read invokes Common Lisp reader, it has some security implications for sure. In my use case however, I didn’t really bother about this part, because I’m a sole user of the cl-journal and database reading and writing logic is written in such a way, that I can change it to whatever format anytime without any friction.

Another usage was showcased in part about html to markdown conversion. Serialization logic in plump simply prints results in *stream* variable that by default happens to be equal to stdout (if I’m not mistaken) and in the method I could capture any part of output simply by temporary setting *stream* to the other stream and then doing whatever I found fit with the output.

(with-output-to-string (out)
  (let ((*stream* out))
    (loop for child across (children node)
          do (serialize-object child))))

format works with streams too, I’ll talk about it later.

Special variables

Special variables is yet another super powerful concept. What’s special about them is that they use dynamic binding instead of a lexical one and that means the changing the value of such not only for the current scope but for all the code that is called from there.

This gives a powerful weapon to penetrate through layers of abstractions without a cost of passing this variable through or making it global in the true sense of this word.

In case of plump special variable provided an easy way to control the output.

Loop

If were to write this post a couple of months ago I wouldn’t mention loop at all. I came to Common Lisp from clojure and my brain was really fixed on immutable data structures, hence I tried to avoid any imperative constructs. Another moment was that loop had a controversial perception across community some of which preferred iterate and the rest tried to avoid both.

The turning point was when I started writing fetch and sync logic and I had to iterate there a lot in many different ways. I tried loop once then twice and then I ended up using it all over the place. Why so?

Simply because loop unify all different looping constructs that other languages have like for or while or even map in one unified call and in addition to that gives local bindings all an easy way to execute body only under certain conditions and return from any point, and I’m sure there are lots of things I’m not aware of.

Let me show a couple of especially impressive examples from the code. Here is a function that prints a list of files that need to be merged:

(defun get-merge-candidates (db)
  (let ((store (restore-source-posts (fetch-store db)))
        (ht (to-hash-table *posts*))
        (visited (make-hash-table)))
    (loop for event in (events store)
          for itemid = (getf (getf event :event) :itemid)
          for post = (gethash itemid ht)
          when (and (not (gethash itemid visited))
                    (or (not post)
                        (older-than-p post (getf event :sync-ts))))
            collect
            (progn
              (setf (gethash itemid visited) t)
              (if (null post)
                  (format nil "~a - ~a"
                          itemid
                          (getf (getf event :event) :url))
                  (format nil "~a - ~a (~a)"
                          itemid
                          (getf (getf event :event) :url)
                          (filename post))
                  )))))

Iteration goes over (events store), two other fors work just like local bindings. when part uses local bindings as well as bindings from function scope to understand if this particular post needs to be merged. If this check succeeds collect executes next form and appends it to a resulting list to be returned. In this body we update a list of visited posts so that any duplicate post is ignored.

Next example is silly but show loop can serve as a loop construct.

(defun lj-get-server-ts ()
  ;; scary hack to get server ts in a single timezone
  (labels ((r () (getf (lj-getevents '(1000000)) :lastsync))) ;; something big enough to have empty lookup
    (loop with a = (r)
          do
             (let ((b (r)))
               (format t "~a~%" b)
               (when (older-p a b 10) (return a))
               (when (older-p b a 10) (return b))))))

with serves as a local binding there and whenever we want to finish we can simply call return and it’s argument will be the return value.

Yet another loop combo allows an easy traversal of plist with destructuring of key/value pair:

(loop for (key value . rest)
        on (getf post-file :fields)
      by #'cddr
      do
         (format out "~a: ~a~%"
                 (string-downcase (symbol-name key))
                 value))

From examples above you can already spot the greatest weakness of this macro - it’s syntax is so diverse that using it looks natural only when you read it, writing tends to be more in the field of trial and error.

Macros

There is a lot written about macros and their pros and cons. Main drawback for me is that their usage or, better, usage of nonstandard ones has a huge impact in readability of the code simply because you need to go and understand them first and macros are not the easiest thing to understand, especially for people without years of full time lisp development like me.

From the other side writing them is a total pleasure because you can watch the language mold under your hands. I’ll show one example there and it’s naming is probably totally incorrect, but it solved the issue I had.

To print the status I had to print information about different states - print a list of new files, drafts, updated files etc. Each of them had it’s own sub to produce a list and it’s own text of course. To make it more human-friendly I wanted to have not one but three text - for zero, one and many results.

I wrote initial logic as a function and amount of duplication became obvious very soon. I decided to give macros ago and imagined a perfect way to generate a status of a certain kind. I could do a macro that does code execution in place but that would sit in the body of a single function inflate it’s size. Based on that I decided to make a macro to generate a function definition.

This was a perfect syntax in my opinion:

(with-files new (get-new-files)
  "There are ~a new files to publish~%"
  "There is a new file to publish~%"
  "No new files to publish~%")

That would generate a function with the name with-new-files that takes a callback to print a list that’s a result of calling (get-new-files) form and prints header by itself and items list with this callback.

The callback was an extension point to remove any constraint on the type of list items and let actual code deal with it.

With the macro implemented actual status code again became really trivial.

(defun print-status ()
  (flet ((print-names (items)
           (format t "~%~{    ~a~^~%~}~%~%" (mapcar #'filename items)))
         (print-string-names (items)
           (format t "~%~{    ~a~^~%~}~%~%" items))
         )
    (with-draft-files #'print-string-names)
    (with-new-files #'print-names)
    (with-modified-files #'print-names)
    (with-deleted-files #'print-names)
    (with-fetched-files #'print-string-names)))

And here is the actual macro code:

(defmacro with-files (name accessor multiplemsg singlemsg nomsg)
  (let ((fn-name (intern
                  (concatenate 'string
                               "WITH-"
                               (symbol-name name)
                               "-FILES"))))
    `(defun ,fn-name (cb)
       (let ((items ,accessor))
         (if (> (length items) 0)
             (progn
               (if (> (length items) 1)
                   (format t ,multiplemsg (length items))
                   (format t ,singlemsg))
               (funcall cb items))
             (format t ,nomsg))))))

What happens there is I’m assembling function name and then return function definition with placeholders replaced with the data from arguments.

Macro is not a tool for any task but it can be really life changing if you only ever programmed with macroless languages.

Format

I find format really fascinating. Most of the languages mimic C, they implement sprintf style functions and it’s a shame given how much more powerful format is.

First of all, it works on streams, that gives a lot of power on their own, as I wrote earlier. Next it literary has all functionality necessary to always print to output with one call.

It’s possible to print a quoted value, a value without surrounding quotes, add all sorts of paddings to the printed data and to even print arrays!

Here are few spells:

(format nil "~{~a/~}~a-~2,'0d-~2,'0d-~a.md"
        rest
        (getf date :year)
        (getf date :mon)
        (getf date :day)
        name
        )

What happens there? We print a pathname. rest contains a list with folders, date parts go after that and in the end we want to print the name. ~{~a/~} prints folders separated by the forward slash, ~2,'0d ensure that two digit number is printed and pads it with leading zeros if necessary. If arguments are ((list "path" "to" "file") 2018 3 4 title result will be path/to/file/2018-03-04-tile.md

Or here is how I print a list of files in status, ~% means newline:

(format t "~%~{    ~a~^~%~}~%~%" items)

Format can do much more and it simply eliminates manual fiddling with data to prepare it for printing.

Little things

There are lots of decisions in Common Lisp standard package that make you only wonder why didn’t the find the way to any other language.

For example, read-line function accepts an argument that will be returned if the end of stream was reached. Why would you need this? Simply because returning custom value there can make some upper-level logic more generic. Btw, dolist does that too.

Another small nicety is that many functions on collections accept parameters like :test or :key that immediately make them more useful.

Here is how last published post is found, for example:

(defun get-last-published-post (db)
  (car (sort (posts db) #'> :key #'created-at)))

Not effective, I know. Or member function in this example:

(defun known-editor (e)
  (member e '("vim" "emacs" "emacsclient") :test 'string=))

It’s string there, but having this parameter immediately allows to have list members of any type as long as there is an equality check for them.

Common Lisp pitfalls

Packages

Before writing cl-journal I didn’t have too much experience working with Common Lisp, so I decided to pick up whatever I considered to be latest best practices and try to live with it. I’ve split all the functionality into separate packages and tried to export and import only really necessary functionality.

Soon I’ve found out that this way of development was much more verbose there than in other languages and the reason was mostly CLOS. Packages serve as containers for symbols in Common Lisp and any symbol you want to share between packages needs to be exported.

That means that class name should be exported as well as all generic functions generated for it accessors. A good example is cl-journal.db package. Three classes with a handful of slots each generated a long list of symbols to export and besides that, there were still ordinary functions and special variables. And since generic functions where magically generated it still left open questions about how they will work if such generic functions were imported from two different packages into third one. I’m sure this behavior is defined somewhere, but I don’t know.

In the end, I got so tired of all this maintenance that I started importing whole packages with :use even though I treated that earlier as a non recommended way.

Standard library

I think a very thick book can be compiled from all the complaints regarding the standard package of the language. Sometimes there are functions that are of no interest to most people and there are many cases when obviously necessary functions are not there.

Good examples are user input, string manipulations or external processes. Some things like prompt are more or less easy to do, but password input proved to be a really painful exercise. uiop/run-program is also far from the easiest function to use. Ok, maybe I missed a very good tutorial on this one, but I had to go through it’s code several times to understand the details or meaning of it’s parameters.

Common Lisp comes from the time when it was if not mainstream but a widespread language with lisp machine legacy and from what I understand this had an influence on it’s relations with outside world and that really hurts, especially comparing string and io operations with languages like perl that do that this bit particularly well.

Third-party libraries

That’s another very common complaint. Common Lisp libraries are often of fantastic quality feature-wise but it doesn’t really help if they have no documentation or a brief one that explains 10% of the functionality. And you can almost forget about library specific tutorials. Getting over this was a rewarding intellectual achievement for me but the price was time, lots of time.

Here I’d like to admit that plump library had one of the best documentations and even a couple of projects using it in neighbor repos. This bit really helped me in understanding the library and coming up with a proper solution for html conversion.

Final thoughts

You probably spotted already that I’ve been mentioning Livejournal everywhere it can give an impression that there is no way in life for the client to support any other service. That’s actually not true, but will require a good amount of work of course. The first step would be to abstract away remote api and the second one will be to make <post> and <post-file> classes service agnostic. It’ll be even easier if we do not set the aim to support the same feature set for every single platform and provide a cl-journal as a platform that can give the same experience as it does now for Livejournal with service specific changes.

In terms of featureset, a lot of things can be done better of course. For example, I can think of template posts, or client can support failures much nicer, but after using it for more than two years I can say that none of it is something that turns it into something unusable.

Was it a correct choice to use Common Lisp for implementation? For me, absolutely yes, because a lot of things I implemented would have taken twice as code to get them working and interactive development mad me so performant that I was able to add significant features to the code even with a very limited time I had.

One good learning for me was to add test into interactive development workflow. It was really not that obvious for me, but comment driven development as I did for syncitems with an addition of test allowed me to write functional code even in the time of heavy sleep deprivation that I tend to fall into.

Another good learning that I had was to follow a bottom-up approach, you could see an example in the merge description. Whenever I approached big task I started asking myself very simple question starting from “How do I get the changes?” or “How do I get the filename” and solving them slowly one by one. After they were all complete the final task that was thought to be complex and indeed was turned out to be a very small function with simple logic.

Thank you for reading.

Part 4: Merge

2018-06-28T00:00:00+02:00

With a database of raw posts at my hands, I started thinking about ways to implement the merge. First and foremost I had to identify which posts actually had to be merged. This is trivial with missing posts, but for existing ones I had to have something to compare them to, and since I had raw posts from the server with the server timestamp I had to get server timestamp for all existing posts and update it for any posts that I wanted to change.

If you remember I already had ignore-all command to bump local modification timestamp of all posts and I ended up doing the same for the remote timestamp. The easiest way to get it was to take it from the syncitems call and to crossect them with all existing posts. As a first step, I ensured that every fetched post received corresponding syncitems update timestamp and after that I only had to add this timestamp to all the posts in the database.

And this is where my obsession with loop macro began.

(defun mark-as-pulled ()
  (let ((ht (-<> *posts*
                 (fetch-posts)
                 (restore-source-posts)
                 (to-hash-table))))
    (loop for post in (posts *posts*) do
      (let ((ts (gethash (itemid post) ht)))
        (if (null ts)
            (format t "Item ~a was not fetched yet, run cl-journal fetch first~%" (itemid post))
            (setf (server-changed-at post) ts))))
    (save-posts)))

You see there old friends - -<> macro and to-hash-table function which became independent implementation there with an option to chose what value to use as a key for a hash table. Once that was done update became as trivial as setting slot values for any posts that were found in the hash table.

That is one direction, but as I said there was another one - I had to get a server timestamp for any post update or creation I did. The post data being returned contained some timestamps but none of the public ones contained a timestamp of last post update. I started looking elsewhere and found that getevents call also returned server timestamp. Super handy! I did a quick function to get a timestamp based on the call only to find out later that timestamp did not always match the one from the syncitems.

What the hell? Timezones. Livejournal api is quite naive in terms of dates in a sense that all the timestamps it returns are in format YYYY-MM-DD HH:MM:SS and one can only assume that they all relate to the same timezone and getevents call was one particular example where this invariant didn’t hold, or to say it more precisely, not always. It seems like this api call is served by the two servers in two different dcs with their local timezone set. syncitems seemed to work with UTC timezone and I started looking around in search of something that can work. First solution I came up with was rather horrible: we just need to keep calling getevents till we get two different timestamps and from that point we can take a lower one and save it along the post.

(defun lj-get-server-ts ()
  ;; scary hack to get server ts in a single timezone
  (labels ((r () (getf (lj-getevents '(1000000)) :lastsync))) ;; something big enough to have empty lookup
    (loop with a = (r)
          do
             (let ((b (r)))
               (format t "~a~%" b)
               (when (older-p a b 10) (return a))
               (when (older-p b a 10) (return b))))))

You see? It’s loop again. What happens is that we keep calling the api and keeping the result of the previous call. Once we see the difference we return the earliest. Loop works wonderfully there: it allows to set a local binding a and then execute loop body and return from it with return.

older-p is also interesting. I didn’t find a simple way to compare YYYY-MM-DD HH:MM:SS timestamps, however, local-time library provided a way to compare it’s time object instances. And this helper function exploits this fact:

(defun older-p (ts1 ts2 threshold)
  (labels ((parse (ts)
             (let
                 ((local-time::*default-timezone* local-time::+utc-zone+))
               (parse-timestring ts :date-time-separator #\Space))))
    (timestamp>
         (timestamp- (parse ts2) threshold :sec)
         (parse ts1))))

Setting timezone doesn’t have any special meaning there, it’s just some stable value that allows us to compare timestamps. The threshold is there to filter out small inconsistencies between time stamps.

While it worked at the beginning, later I was punished for my naiveness and getevents started returning mostly non UTC datetime. I decided to search more and finally understood that I could as well just use unix timestamp that some of the calls return and then format it according to timezone of my liking. local-time library was really helpful in this sense. Timestamp formatting can be implemented as easy as:

(defparameter *livejournal-timestamp-format* (list
                                 :year
                                 "-"
                                 (list :month 2)
                                 "-"
                                 (list :day 2)
                                 " "
                                 (list :hour 2)
                                 ":"
                                 (list :min 2)
                                 ":"
                                 (list :sec 2)
                                 ))

(defun format-unix-timestamp (ts)
  (format-timestring nil (unix-to-timestamp ts)
                     :format *livejournal-timestamp-format*
                      ;; looks like utc zone is what syncitems use
                     :timezone +utc-zone+))

Initially I used defconstant there, but every copilation resulted in the restart and sbcl even mentions this behaviour. I gave up and moved on to lj-get-server-ts function that now looked much simpler:

(defun lj-get-server-ts ()
  (->
   (rpc-call "LJ.XMLRPC.getchallenge")
   (getf :server_time)
   (format-unix-timestamp)))

After this was done I had a clear way to get posts to merge and it appeared to be an interesting exercise! Fundamentally speaking there are two parts: first, we need to map all the fields supported by the client to the fields in a raw response, and second, we need to take post body from the response which happens to be html most of the time and convert it into markdown.

Fields

Livejournal API is really thought through for it’s age and has a lot of decisions make it handy for the clients. One example is a changelog provided by syncitems call. The other one is that although it’s not REST or anything alike original authors already thought about entities and it appeared that fetch api returns more or less the same structure that post api accepts. More or less means that main fields are on the same places, however, reading entity returns more data and most of it was added much later and is not documented anyhow. Another important bit that whenever field contains non-latin characters it’s accepted as is for posting but is returned encoded in base64.

I didn’t know or care about this similarity when I wrote posting part and didn’t spot it right away when I did reading part and that led to the existence of two different pieces of code to handle fields. About the first part I already wrote, see the description of to-event-list generic. Reading part is handled by parse-xml-response in file-api. The idea of this function is to return parsed data in such a way that I can use it later to create instances of <post-file> of <post>. Potentially non-latin fields are handled by b64getf function that works almost like getf however, if received value is a list with car equal to :base64 it takes cdr and decodes it.

Markdown

Now, one more interesting part, at least it was really interesting to me to implement. The most important part of the raw post was an actual post content. And it was returned mostly in the form of html both for posts written via Livejournal web interface and with the client (which was expected since client converted markdown to html before sending).

Given that what I wanted to get was to convert noncomplex html like this:

<h1>One more heading</h1>

<p>First <em>paragraph</em></p>

<blockquote>As someone said</blockquote>

to a pretty markdown form such as this:

# One more heading

First *paragraph*

> As someone said

To complicate things a bit more Livejournal had a couple of custom tags in use. The trickiest one was lj-embed that was created for any embed from youtube or a number of other services. At first problem looked unsolvable there, because by default response contained only tags like <lj-embed id="1"> which gave no chance to guess the contents. Luckily Livejournal once was an open souce platform and a number of people made snapshots of the source code before it got closed. After a thorough inspection of xmlrpc api implementation I found that there was a magic undocumented flag get_video_ids which allowed to add a bit more information to the lj-embed and turn it into something like <lj-embed source="youtube" vid="1bU7COLWJE0"> which was totally enough to restore original embed code.

In order to put that together, I looked for an appropriate html parsing library and after a couple of experiments it turned out that plump library was completely sufficient for the purpose. The trick is that plump can not only parse html but also serialize it back and this is where it appeared to be super easy to plug in and turn html generation into markdown generation. Serialization is actually done by one generic function, plump-dom:serialize-object and Common Lisp generics and auxiliary methods made it trivial to achieve desired behavior.

First, I was able to target only tags I was interested in by specifying the type of the first argument, which represented the element. After that, by specifying :around method I could completely control printing behavior. Let’s say that we want to implement only paragraph, links conversion and lj-embed support. Here is how the code would look like (you can check [full source][markdowwnify].

(defmethod plump-dom:serialize-object :around ((node element))
    (cond
      ((tag-name-p node "P")
         (loop for child across (children node)
               do (serialize-object child))
         (format *stream* "~%~%"))

      ((tag-name-p node "a")
       (let ((str (string-trim '(#\Space #\Newline)
                               (with-output-to-string (out)
                                 (let ((*stream* out))
                                   (loop for child across (children node)
                                         do (serialize-object child)))))))
         (format *stream* "[~a](~a)" (if (> (length str) 0)
                                         str (attribute node "href"))
               (attribute node "href"))))

      ((tag-name-p node "lj-embed")
       (cond
         ((string= (attribute node "source") "youtube")
          (format *stream* "<iframe width=\"560\" height=\"315\" src=\"https://www.youtube.com/embed/~a\" frameborder=\"0\" allow=\"autoplay; encrypted-media\" allowfullscreen></iframe>" (attribute node "vid")))
         (t (call-next-method))))

      (t (call-next-method))
    ))

What happens is that we think of every paragraph as of a block of text with two newlines after it and this is what happens. By calling serialize-object we pass control back to generic and recursively it can return back to this method to do any conversions we’re interested in for the child nodes of the paragraph.

Also, another awesome part of this api is that serialize-object doesn’t return a string, it prints to a stream and that means that we don’t need to waste our time with doing string concatenation and related housekeeping but we are still in full control of the fate of the content to be printed.

Link conversion shows it very good. I override *stream* stream that plump uses for printing and catch all the children output into a string, which gives me a way to produce a link like [contents](url).

In this sense, lj-embed conversion doesn’t show anything new, but I decided to add it to show how easy it was.

Final conversion code turned out to be more or less lengthy but that’s only because I was already too tired and lazy to come up with smart solutions and the only thing I cared about was to make enough test cases and make them pass.

One library that I found super useful at this step was cl-strings that provides basic string manipulation utilities that all languages except Common Lisp provide out of the box. Of course, I could have used cl-ppcre but cl-strings just did the job and again made the logic trivial.

I can say that it was magically simple for me due to how beautifully different Common Lisp played together in api. And plump is awesome by itself!

Filename

Once I had all proper fields and post source in place the last remaining part was to actually save the file. For existing posts it’s easy, but for new ones I had to come up with a name. When I write a post with a client, cl-journal new post-name command generates filename YYYY-MM-DD-post-name.md. post-name was completely arbitrary and for reverse action I decided to generate it from the title. I almost decided to write such a library myself but then stumbled upon [reddit post][reddit slugify] where author presented cl-slug library which solved exactly this problem and not only for English but also for a number of other languages including Russian. Thank you André!

After slug was there I had to ensure the uniqueness of the filename. final logic turned out to be pretty straightforward: generate base in the form of YYYY-MM-DD-slug where timestamp was creation timestamp of the post and if there was already a file with the name YYYY-MM-DD-slug, keep adding -1, -2 etc at the end till we found a vacant spot.

Here is a full code of the sub:

(defun generate-unique-filename (db datetime base)
  "Generate a filename that does not yet exist in the database
   based on datetime (yyyy-mm-dd hh:mm:ss) and a base (any string).

   Function will generate a base name of the form <date>-<slug>.md
   and in case such a filename already exist, will keep adding increasing
   postfix numbers till it finds a vacant spot"
  (labels ((gen-name (date slug counter)
             (if (equal counter 1)
                 (format nil "~a-~a.md" date slug)
                 (format nil "~a-~a-~a.md" date slug counter))))

    (let ((date (car (split datetime)))
          (slug (slugify base))
          (ht (to-hash-table db :key-sub #'filename)))

      (with-output-to-string (out)
        (loop with cnt = 0
              for name = (gen-name date slug (incf cnt))
              while (gethash name ht)
              finally (format out "~a" name))))))

Did I just use loop again? Oh, my. And to-hash-table was useful again with a little modification of :key-sub parameter that allowed to have anything from the post as a key. In this case, using filename as a key made generation logic really simple.

Now I have all parts in place - all posts could be fetched from the server, I could determine which posts were updated remotely or just didn’t exist locally and I could convert remote representation back to the markdown text (I omitted the code that turned fields into a header with title: and other fields but it’s there of course.

By the way, in case usage of the format in finally part surprises you, I had to add it, because otherwise, sbcl kept serializing filename as an array of characters for some reason.

merge-fetched-posts function contains one of the most epic usages of loop macro I ever did. Let me just paste it as an excerpt from the sub, full code is there:

(loop for event in (reverse (events store))
        for itemid = (getf (getf event :event) :itemid)
        for post = (gethash itemid ht)
        when (and (not (gethash itemid visited))
                (or (not target-itemid)
                    (equal itemid target-itemid))
                (or (not post)
                    (older-than-p post (getf event :sync-ts))))
        do
        (stuff))

All the logic fit nicely in default features of loop. reverse it’s there because I wanted to convert only the most fresh version of the post that I downloaded, for part of the macro played the role of let and assigned values on every loop iteration and when included all the logic necessary so that loop body executed only in case it had to: in case post was not already generated during this run, it didn’t exist in database or database version was older than remote.

One important thing in this logic was synced-from-fetch flag for every post. Whenever I merge it I raised this flag to 1 and whenever I updated or created a post with the client I reset it to nil. Why important? Simply because the conversion process was and still isn’t perfect, I wanted to have a nondestructive way to run merge functionality again and again, end this is what remerge-fetched-posts function and related cli command does. With this flag I could take all the posts that I already converted but that were not touched after and merge them again. This gave me an opportunity to already have markdown files of some sorts and still work on better conversion afterward.

From potential improvements:

plump doesn’t handle accidental closing image tags as I wanted them to (I want them to be ignored)
All paragraph contents tend to be turned into one line and in a perfect world I wanted my converter to also have line length limitation and nicely wrap the code if necessary.
And probably many more bugs and mistakes that I did myself.

After all this functionality got implemented, the client reached near perfect state to me, because if you remember it’s unidirectional nature was one of the client limitations and now it actually didn’t matter where a post was written or updated, fetch end merge steps could incorporate all the changes back. And what’s even cooler, now the client can be used for any blog and from the day one it’s possible to get all posts offline and work as if you were using cl-journal forever.

Part 3: Fetch it!

2018-06-25T00:00:00+02:00

All the functionality mentioned before was quite complete and I used it constantly for a year or even more. There were two drawbacks, however:

I had source code only for the posts I wrote using the client. Nothing from earlier days was preserved and in case livejournal.com goes down forever, I would lose them all.
All the posts had to be written and update later with the client, because it had no way to know about changes done elsewhere.

Both parts were not important for day to day writing activities, but the former kept returning after every next negative news about the service and the latter was more of a challenging task that I wanted to tackle. I resisted for a long time but finally gave up and decided to write the logic. Besides that, how could it be a real git like client if it has push command but didn’t have fetch or merge?

For the first question fetch was the most important bit. I assumed that if I downloaded posts in any form or shape I could work later to turn them into a markdown later without any hassle.

Luckily original Livejournal authors assumed such needs and designed API protocol to enable it (Kudos to Brad Fitzpatrick!).

First important bit is syncitems. What it allows to do is to get updates since some timestamp or from the beginning. A change can be a new post, an update to post or a comment, and since any post could be updated several times, duplicates were expected. The only thing protocol did not support were post deletions, which meant that I wouldn’t notice such a change.

I didn’t care about comments or deleted posts, however, what I wanted to get was a list of posts that changed somehow since timestamp. To make things even more complicated, api call didn’t return all changes but just some subset of it and I had to do several calls in a row to get the desired result.

Logic looked quite complicated and I ended up literally writing the steps algorithm should take and them slowly implementing them step after step and surrounding them with tests to keep it under control, you can check the final logic in get-fetched-item-ids function in lj-api. As you can see I used -<> threading macro all over the place to keep the code readable. Apart from the logic itself, I ended up implementing some function helpers that I didn’t find in the language but that we’re very useful.

The first example of this is acc. I do a lot of Perl and data traversal is really good there. What the language allows you to do is to easily access nested arrays without checking the existence of anything - $obj->{a}{b}{c}. Common Lisp getf is much less flexible and allows only one level lookup and this is where acc is useful since it allows any series of keys:

(defun acc (l &rest args)
  "Access member in a nested plist.

   Usage (acc l :one :two :three)"
  (cond
    ((null args) l)
    ((not (listp l)) nil)
    (t (apply #'acc (getf l (car args)) (cdr args)))))

Another two functions are partial and print-and-return. Name of the one first is self-explanatory and the next one accepts a value, prints it and immediately returns it. This is useful in the middle of threading macro call, because it allows printing intermediary steps in the computation.

One last perlism is to-hash-table. In Perl transformation between a list and a hashmap is extremely simple and happens all the time. This is how I would do it:

my %ht = map { $_->{id} => $_ } @list;

This simplicity means that most of Perl code consists of jumps between lists and hashes to use whatever works best in a particular situation. Posts are stored in the database as a list hence I had to iterate over it in a more or less smart way all the time. A list is a reasonable structure there and one of the ways to fix the problem with lookups was to maintain a hashtable in the class, but I I decided to go with a simpler one and convert the list to the table on the go.

(degun to-hash-table (l)
  (let ((ht (make-hash-table :test 'equal)))
    (dolist (item l ht)
      (setf (gethash (car item) ht) (cadr item)))))

This snippet highlights one of the neat features I like so much in Common Lisp. Maybe it has not the best possible standard library in the world, however, there are true gems of usability there. In this particular case, dolist accepts the third parameter which will be returned as a result of dolist. That means that you can write things like filling in hash table very naturally.

After get-unfetched-item-ids function was done, the next one on the list was one to download a list of posts. And here it got a bit tricky because of Unicode.

Unicode and Livejournal

Livejournal is there for quite a long time and that means that it’s codebase and data predates the times when everything became all Unicode. Many veteran coders can still show scars from that time, yeah. In essence, there was an ASCII table that was used to define a meaning of the byte of data (hence single byte encoding) and it defined the meaning of the first 127 values and left the rest undefined ant that was used as a space for extra characters for every specific alphabet. In Russia, there were two most popular encodings - cp1251 and koi8-r. How did it affect Livejournal? As is written in their FAQ, they had no way of understanding the encoding and hence it was left up to users to choose a proper one to render a page.

In case such encoding was chosen, Livejournal API allowed to download both Unicode and non-Unicode posts in the same way. Unfortunately, what I found empirically, serverside encoding did some weird stuff presumably because the text was already converted to Unicode somewhere along the way, and I had to disable it in order to get a properly readable way.

This decision had consequences in a sense that now I couldn’t download different types of posts in one batch - Livejournal api returned an error in this case. The beast I ended up writing is called lj-getevents-multimode you can check it out in lj-api. I baked in a couple of assumptions in the code:

One of the api versions (Unicode one) was much more probable
If the post had one version, the next one had a high chance of having the same one.

Final logic looked like this:

Try to download posts in one version
If fails reduce batch to one post and repeat
If that fails, flip download mode and repeat
If that fails, error out
If one of the previous steps succeeded, increase batch size 2x and repeat.

Maybe a bit naive approach, but it worked really well in the end. After these two bits were done, fetch logic started looking very simple:

(defmethod fetch-posts ((store <store>))
  "Fetch all new items from remote service since last-fetched-ts
   of the store"
  (multiple-value-bind
   (new-itemids last-item-ts ht) (get-unfetched-item-ids store)
   (cond
     ((null new-itemids) store)
     (t (let ((new-events (-<> new-itemids
                             (lj-getevents-multimode)
                             (getf <> :events)
                             (mapcar #'(lambda (x) (enrich-with-ts x ht)) <>))))
          (merge-events store new-events last-item-ts)
          (fetch-posts store))))))

<store> there is another class I created to store a list of downloaded posts in their original form. enrich-with-ts function exploited the fact that get-unfetched-item-ids could now server change dates for any updated post and download post api call did not return that and it simply added such a timestamp to every post. merge-events did no more than placing downloaded posts at the end of the list.

Now, where should I take this store? I mimicked the way I stored and saved the database with posts and added a hacky solution to do lazy loading of the database. I don’t think it’s necessary since posts are downloaded differently, but in case you wonder here is the implementation:

(defmethod slot-unbound (class (db <db>) (slot-name (eql 'fetch-store)))
  (setf (slot-value db 'fetch-store)
        (make-instance '<store>)))

This gets triggered whenever slot doesn’t have value set and function sets slot value after creating the class instance. Next time slot already has value and this method is not called anymore.

Top level fetch function now looked very simple:

(defun fetch-updated-posts ()
  (let ((store (restore-source-posts (fetch-store *posts*))))
    (fetch-posts *posts*)
    (save-source-posts store)))

After I got all this working I got a raw dump of all the posts I ever wrote, which meant that I could safely work on actually converting them back to markdown without fear to lose the contents.

Testing

To be frank sync protocol didn’t come for free to me. Too many moving parts and conditions. And while most of the code base has been written in a pure leisure fashion without a single test, I decided to build new features starting from fetch with at least some coverage. And I chose prove as a framework of choice.

The most annoying bit of this framework is it’s default reporter which uses escape control sequences for colors and emacs requires some additional configuration to make that work and that’s not something I wanted to invest my time in. Instead, I invested my time in finding a way to disable them. It appears that I’ve always been one dynamic variable away from the result:

(setf prove:*enable-colors* nil)

The overall tests integration could have been easier, however, still doable. I’ve made a test system, added a magical spell to the main asd file and test framework was set up. One of the important things to note is that prove itself is a dependency of the test system, hence in order to have it available in the repl this test subsystem should be loaded instead of the main one which will be loaded as a dependency.

A really awesome feature of the prove framework is that it allows to rerun specific tests just by recompiling them. This feature enables near magical workflows when I could prototype a feature and then cover it with tests in real-time without the need to run a full test suite again and again or to do a build every time and run tests there.

Since I wanted to write tests for the logic built around api calls I wanted to mock them to test outcomes of specific sequences of calls. I took mockingbird and it did provide me with a basic feature of mocking any function in any package, however, I ended up implementing a small macro to enable testing a sequence of calls.

Funnily enough, all the mocked calls ended up being trivial, but the possibility is still there! The idea was that if you want to mock a function, say foo and have it return (1 2 3) on the first call and nil on any subsequence you could just write:

(with-mocked-calls
  foo
  (
   (1 2 3)
   nil
   )
   (some)
   (code))

Macro allows generating a lambda function for mockingbird that has a baked in logic that tests against the number of calls and returns a respective result. Here is it:

(defmacro with-mocked-calls (func data &rest body)
  "This function is necessary to emulate the behavior of
   a function that has side effects. Every subsequent
   call to the function will return next item from the
   data list, except the last one which will be returned
   endlessly"
  `(with-dynamic-stubs
       ((,func
         (lambda (&rest rest)
           (declare (ignore rest))
           (cond
             ,@(loop for resp in data
                    for i from 1 to (length data)
                    collect
                    (if (equal i (length data))
                        `(t (quote ,resp))
                        `((equal (call-times-for (quote ,func)) ,i)
                          (quote ,resp))))))))
     ,@body))

I’m no macro guru and I get super excited about every single case when I managed to write something that actually looks like useful thing. You can check the tests for this definition and real-world usage.

Now, let’s merge.

Part 2: Client logic

2018-06-22T00:00:00+02:00

There are a lot of different bits related to the client that were pretty fun to solve, especially in order to get git-like behavior, but nothing of that matters if we cannot communicate with the server, so let’s start with that.

XMLRPC

Livejournal uses XMLRPC as a protocol for communication. Documentation is old but serves it’s purpose. However, I had to check the last opensource version of the protocol implementation to make sense out of it for a back sync functionality. I’ll return to it later, but now let’s see how that can be done at all.

I looked for a library to use and initially my choice was s-xml-rpc. I don’t remember exactly why I decided to get rid of it. I think it was due to the fact that s-xml-rpc returns result in term of a hierarchy of objects and it proved to be difficult to maintain.

Instead of that, I’ve settled with rpc4cl which simply returns a nested list and that’s precisely what I need. Now that I’m writing this, it appears that author removed rpc4cl from Github which is very unforunate. To simplify the code I made a simple wrapper that takes adds host information to every call:

(defun rpc-call (method &rest method-parameters)
  (apply #'rpc4cl:rpc-call *service-endpoint*
         nil nil method method-parameters))

After that, all remote calls naturally fit into the code. Here is as example a sub to get a so-called challenge from Livejournal service:

(defun getchallenge ()
  (->
   (rpc-call "LJ.XMLRPC.getchallenge")
   (getf :challenge)))

Please note -> from cl-arrows here. I seriously cannot imagine my lisp coding with it, because using the arrow prevents all sorts of deep nesting and keeps the code clear. Since it’s not always easy to remember in which place the previous result goes by default in the latest code I always use -<> instead which allows putting a placeholder <> that will be replaced with the result. This way makes argument placing more explicit. As an example getchallenge can be transformed to this:

(defun getchallenge ()
  (-<> "LJ.XMLRPC.getchallenge"
   (rpc-call <>)
   (getf <> :challenge)))

One of the distinctive features of Common Lisp is its live editing process where one can start interpreter, load necessary code there and do all the changes right in it compiling changed functions if necessary without a need to compile and start the program all over again.

In case of the API, it’s particularly useful. I defined three dynamic variables - *service-login*, *service-password*, *service-endpoint*. I set them once the repl starts and after that I can do any experimentation with the api with help of all the power the language provides. And after I wrapped all service endpoints as functions I literally could do all blog manipulations without leaving the editor. It proved to be so stable that I kept the running lisp process for months without any need for a restart.

The only improvement I made recently was to add unit tests into the project development workflow and it works wonderfully with repl-driven development. Why is it cool? With a usual way of developing one of the hardest bits is to recreate an environment where tests happen, and in cases where there is no obvious way to implement something, changing implementation can have a lot of impact on how tests are written. With Common Lisp, I can experiment with code freely until I get something working without overhead related to environment and create supporting architecture only after that and I can still run all the tests right there! Change a function and recompile only one specific test till everything works and then compile the whole suite to see if everything’s in place.

It’s so fascinating that I can safely say that it’s one of the features that really make me stick to the language, I enjoy every second of it.

Storage

I wrote the client primarily for myself and I didn’t want to invest in security more than a secure password storage that was achieved but external tools. I wanted some storage, however.

First things first I decided to have a class that represents a posts database, that looks like this:

(defclass <db> ()
  ((posts :initarg :posts :accessor posts)
   (version :initarg :version :reader version)
   (login :initarg :login :reader login)
   (raw-text :initarg :raw-text :reader raw-text)
   (service :initarg :service :reader service)
   (service-endpoint :initarg :service-endpoint :reader service-endpoint)
   (fetch-store :reader fetch-store)
   ))

That’s the most recent version, of course, the most minimal version contained only posts slot and thanks to the lisp interactivity more slots could be added by class recompilation and all the live instances got new slots automatically.

Anyway, as a next step, I wanted to serialize the database and it’s contents and store it into a file. This is very Common Lisp generics step in. They are completely orthogonal to classes and make it super simple to do all sorts of recursive definitions for particular functionality. In this case, I defined a generic to-list and wrote it’s implementation for the database:

(defmethod to-list ((db <db>))
  `(:login ,(login db)
    :version 2
    :service ,(service db)
    :raw-text ,(raw-text db)
    :service-endpoint ,(service-endpoint db)
    :posts ,(mapcar #'to-list (posts db))))

To make this method work completely I had to only implement this method for post <post> class, which I did.

(defmethod to-list ((post <post>))
  (list
   :itemid (itemid post)
   :anum (anum post)
   :ditemid (ditemid post)
   :url (url post)
   :created-at (created-at post)
   :updated-at (updated-at post)
   :ignored-at (ignored-at post)
   :server-changed-at (server-changed-at post)
   :synced-from-fetch (synced-from-fetch post)
   :log-ts (log-ts post)
   :filename (filename post)
   :journal (journal post)
   ))

Super simple, after that saving database to a file became trivial:

(defun save-posts ()
  (with-open-file (out *posts-file*
                       :direction :output
                       :if-exists :supersede)
    (with-standard-io-syntax
      (pprint (to-list *posts*) out))))

*posts* here is another global var that holds a reference to the open database and *posts-file* is a relative path pointing where the posts file should live.

What happens in the sub is that we pretty print s-expression that happens to be a result of to-list method call into a stream that happens to be an open file. with-standard-io-syntax macro ensures that all the output related dynamic variables are reset to their default state for the time of printing.

After this was done using auxiliary methods made it trivial to save changes on all meaningful actions. I had generics for publishing, updating and deleting the post and all save logic is as simple as:

(defmethod publish-post :after ((db <db>) (post-file <post-file>))
  (save-posts))

(defmethod delete-post :after ((db <db>) (post <post>))
  (save-posts))

(defmethod update-post :after ((db <db>) (post <post>))
  (save-posts))

An obvious downside of this approach is that such an implementation is tied to the class and not to object instance and that can lead into all sorts of troubles characteristic to global state.

Now that the file is saved we need to be able to restore it. That’s also easy:

(defun restore-posts ()
  (read-parse-file *posts-file*
                   #'(lambda (l)
                       (setf *posts* (create-db-from-list l))
                       )))

As you see there are no safety checks there, which might be needed were I to worry about it. I chose for simplicity in this case since for the time being I’m the only user of the program.

create-db-from-list is an ordinary function in this case, not generic, and there is a create-post-from-list for posts. Now, during the development, the structure of the database and posts inevitably changes and we need to handle that somehow and that’s why database has version slot.

What I decided to do was to do database migrations right during the read phase. In order to do that I find a unique feature for the next version of config, do migration and run the same function recursively and by this way eventually get a most up-to-date structure that will be nicely serialized on next save.

(defun create-db-from-list (l)
  (cond
    ((null (find :version l)) (create-db-from-list (migrate-db-v0-v1 l)))
    ((null (find :service l)) (create-db-from-list (migrate-db-v1-v2 l)))
    (t (create-db-from-list-finally l))))

Publishing

Now that I had a database I wanted to fill it somehow with posts! Each post is represented by a markdown file with a header that contains custom fields. For example:

title: This is a cool post
tags: this, is
privacy: friends

# Intro

This post will contain

* A header
* A list
* A couple of paragraphs

Please note that till this moment we didn’t reach any cli interface and git-like state management, hence I wanted to implement a simple function that would take a file, parse it, convert markdown into html and publish it as a post.

Task itself is also recursive meaning that after we parse a file and get a list of fields plus a text we need to parse the result again to map all the fields to a representation Livejournal understands.

The first part you can check out an implementation of parse-post-file function. What it does is it reads header line by line and treats all the first field of the form field: value as individual fields and then all the rest as a markdown body of that post which it converts to the html.

After this step is done an instance of <post-file> is initialized and we can already do all sorts of things with it, not just publishing. For example, we can check if it’s a draft. But let’s publish.

For that we need to convert this object to the format lj.postevent understands. For this to-xmlrpc-struct generic is defined, which is super handy since I can do an implementation for both <post> and <post-file> and use whatever is at hand. For example for a new post it’s always <post-file> and for updates it’s <post>.

For <post-file> I decided to go with a layered approach where I have a basic object and then enrich it with all sorts of additional helpers that contain logic for particular fields. Here is to-event-list which is used by to-xmlrpc-struct:

(defmethod to-event-list ((post <post-file>) &optional (transform #'identity))
  (let ((l (list
            :event (if *raw-text* (body-raw post) (body post))
            :subject (title post)
            )))
    (-<> l
         (add-props (fields post))
         (add-privacy-fields (privacy post))
         (add-usejournal (journal post) <>)
         (add-date)
         (funcall transform <>))))

Every add-* function should return a new object that is potentially the same as l but can be modified version of it. Here is add-usejournal which is used whenever I want to post to a different journal than my own:

(defun add-usejournal (journal plist)
  (if (not (null journal))
      (concatenate 'list
                   plist
                   (list :usejournal journal))
      plist))

This conditional is not that elegant by itself, but the general pattern proved to be very useful and I used it in many different places.

Once I got an event in Livejournal view of it the publish function itself becomes really simple:

(defmethod publish-post ((db <db>) (post-file <post-file>))
  (set-credentials db)
  (let ((*raw-text* (raw-text db)))
    (let ((post (create-new-post post-file)))
      (push post (posts db)))))

And if you remember for the previous parts database will be saved to file whenever this method is called. And now, if we call this method from pre-commit hook we will get all changes saved to disk and will be able to add them to the index and be included in the commit. set-credentials here is to ensure that we have all api variables set before we make a call. The most interesting part of it is get-password that not just retrieves the password, but it also requests and saves it in case of absence.

And this is it about general client framework. With a list of <post> objects in the database we can do all sorts of calculations and answer to all sorts of interesting questions especially because there is a direct connection with the corresponding file on disk that can be converted to <post-file> at will.

How can I get the last published post? Easy:

(defun get-last-published-post (db)
  (car (sort (posts db) #'> :key #'created-at)))

How can I know the title of particular <post>? Super easy:

(defmethod title ((post <post>))
  (title (read-from-file (filename post))))

And that’s just two examples, it’s always possible to start the repl, load a posts database from file manually and start playing with contents.

To make repl experience even more pleasant Common Lisp provides print-object generic that is responsible for the text representation of the object. Hence we can print any information we like from the object instance.

(defmethod print-object ((post <post>) stream)
  (format stream "<post filename:~a url:~a>~%" (filename post) (url post)))

The last bit before implementing cli interface is the state management and it can also be easily implemented using the same operations on <post> and <post-file> objects.

What do I mean by state management? Looking at the blog folder we should be able to understand which posts are new, updated deleted or are drafts and they all are defined in different terms:

New post is the one that exists in the folder but is not mentioned in the database
Deleted post is the one that does not exist in the folder, but does exist in the database
Modified post is that one that exists in both places, however, it’s recorded timestamp is lower than last modification timestamp of the file on disk
The draft is a file in the folder that has a field draft: 1 in the header

What they all share is that they all have a function that is able to generate a list of items of a specific kind and a set of messages to be printed depending on a number of items got in the previous step. Of course, the line can be printed differently from case to case.

For example, we can have the following output:

There a drafts file

    2018-06-04-books29.md

There is a new file to publish

    2018-06-06-test.md

There are 2 modified files to update

    2009-10-12-no-title.md
    2013-12-31-eshe-odin.md

There is a deleted file to unpublish

    2011-01-23-no-title.md

Let’s take drafts as an example. Here is a function to retrieve a list:

(defun get-draft-files ()
  (->> (get-markdown-files)
       (remove-if #'(lambda (fname) (get-by-fname *posts* fname)))
       (mapcar #'read-from-file)
       (remove-if-not #'draft)
       (mapcar #'filename)
       ))

cl-arrows rocks again there. What happens is that we get a list of all markdown files in the folder, remove any files that exist in the database, parse the reminder, kick out all files that do not contain the mentioned field and retrieve filenames back from filename slot in resulting <post-file> objects. Not that performant, I know, but it doesn’t really matter on this scale and my main aim was correctness and readability, not speed.

Once we have a list, we need to have a pretty printer for it and it should be similar for all the cases. At this point in time, I decided to test my defmacro-fu and came up with a macro, that looks like this:

(with-files draft (get-draft-files)
  "There are ~a drafts files~%"
  "There a drafts file~%"
  "No drafts found~%")

What this macro does is it generates another function named using the second argument (it’ll be with-draft-files in this case) and that function will print one of the strings depending on the number of items from the second argument ((get-draft-files) in this case) and print a list of items in a specified manner. The resulting print-status function looks very clean after all these manipulations:

(defun print-status ()
  (flet ((print-names (items)
           (format t "~%~{    ~a~^~%~}~%~%" (mapcar #'filename items)))
         (print-string-names (items)
           (format t "~%~{    ~a~^~%~}~%~%" items))
         )
    (with-draft-files #'print-string-names)
    (with-new-files #'print-names)
    (with-modified-files #'print-names)
    (with-deleted-files #'print-names)
    (with-fetched-files #'print-string-names)))

I use one or another printing function depending on what is produced but the with-* function, it can be a plain list of filenames or a list of <post> instances or anything else and I don’t need to worry about it at the generation step and have all the flexibility to do it at printing step, view layer is separate!

Speaking of modified posts. Whenever I publish a post I run get-universal-time and record it along the filename. Then it takes a super simple predicate to check whether a particular post is modified:

(defmethod modified-p ((post <post>))
  (let ((fname (filename post)))
    (and
     (probe-file fname)
     (< (or (ignored-at post) (updated-at post) (created-at post))
        (file-write-date fname)))))

You see file-write-date function there. This logic works really well if no one touches files, however, whenever files get moved or repo gets cloned to another location, the timestamp from it gets distorted and all files are marked as modified. This is precisely the reason for ignored-at field there. Whenever I want to mark all the files as uptodate I run this function:

(defun ignore-all ()
  (let ((ts (get-universal-time)))
    (loop for post in (posts *posts*) do
          (setf (ignored-at post) ts))
    (save-posts)))

This was one of the first times I used loop macro and I really fell in love with it some time after that.

Markdown

Last fun bit from the basic implementation is about markdown formatting. I decided to use it from the very beginning simply because it’s an order of magnitude better to write than html and still allows to have markup in place comparing to real plain text.

And of course, both vim and emacs have excellent markdown modes and make it a real pleasure to write.

What could be interesting about writing markdown? Well, to extend it!

I had two extensions planned. First of all, I didn’t want to use real urls for links between the post. If I did, I would be tied to one particular web service and migration to another one would be error-prone. If all links in the source code point to other markdown files, that means that whenever I decided to republish all posts somewhere else, I could have all the links resolved.

For markdown rendering I used cl-markdown library and I’m not sure if it was the best choice possible. Why? Because it’s source code is close to impenetrable to me and documentation is almost non-existent. Maybe I’m not a good reader, but I took me several evenings to get my head around library internals and to understand ways to extend it.

For links generation I kept the assumption that I only have one database open at a time and it live in *posts* variable. cl-markdown unfortunately doesn’t export any methods it uses for html generation and I had to get into library package to extend it the way I want.

Inline elements are rendered using render-span-to-html generic function and Common Lisp rocks again there because it allows adding an auxiliary method for it and target a specific set of arguments, that allows to keep method body clean from unnecessary modifications:

(defmethod render-span-to-html :before
    ((code (eql 'inline-link)) body encoding-method)
  (let ((record (cl-journal.db:get-by-fname cl-journal::*posts* (cadr body))))
    (if record
        (setf (cadr body) (cl-journal.db:url record)))))

What happens is that for links the first element in the body element holds a link that will be put in href, and in before element we can do a lookup in the database and replace a link if necessary. I think it’s an amazing level of flexibility with this amount of effort.

Another extension that I wanted to have was about links to the other blogs. Livejournal has some custom tags for different purposes and in this case it uses <lj user="can3p">, where user attribute holds the name of a blog. cl-markdown allows extensions and in order to make {lj-user can3p} render to the desired tag it’s possible to use an official extension logic:

(defextension (lj-user :arguments ((name :required)) :insertp t)
  (setf name (ensure-string name))
  (let ((safe-name (html-safe-name name)))
    (ecase phase
      (:parse)
      (:render
       (format nil "<lj user='~a'>" safe-name)))))

This looks and is indeed easy but it took me quite some time to figure all the details out.

This part concludes main highlights of the client. Of course, there is more, you can check out the amount of commands supported at the moment. Some of them like new or last are there just for convenience to make it easier to start writing a new post or quickly open the last one in case of typos, some like status or url are informative in a sense that they help to query the database and give out information, last group like ignore-all is used to fix the imperfections in file state handling logic.

There is one more group I haven’t talked about, which is fetch, merge and remerge and this is what I’m going to talk about next.

Part 1: Intro

2018-06-19T00:00:00+02:00

Once I decided to sit down and write a perfect blog client for me. Since it was not just about the result but also about the process I chose a language I enjoyed the most and this is how it ended up being written in Common Lisp. I didn’t think I’d get that far and I’d like to share my experience in the process along with all little details that I had to go through during the implementation.

Disclaimer Please treat this post as an experience report more than anything else because this is what it is. As a consequence, there can be a lot of things real lisp wizards will raise an eyebrow on.

Note 1 I’m glad to hear any feedback. If you have any suggestions on the text, want to fix a typo or correct my explanation, please consider opening an issue in the blog repo. If you think that client code doesn’t do what it has to do, please make a pull request or open an issue in client repo.

Note 2 I decided to make all links to the code against fixed sha1 so that future changes do not invalidate them. I’m not sure if I will frequently revisit this document and I hope that the post will remain relevant with this approach.

Below you will read a story about writing a blog client for livejournal.com web service in Common Lisp. As one may note, it sounds like using a very niche language for a near dead web service. To make you a bit more surprised I can add that I’ve been working on it on and off for a couple of years now.

It used to be very simple but now cl-journal is quite powerful. For example, it handles:

Any Livejournal compatible service (like Dreamwidth)
Publishing new posts and updating changed ones
Post text is written in markdown, many Livejournal fields are supported, such as visibility, location or music.
Fetching posts from remove server and transforming them into markdown format.
Lot’s of smaller stuff like links to local files being transformed to actual post urls, drafts, pre-commit hooks, password management etc.
And a lot more!

I’ve been using Livejournal since 2005 and never had a reason to switch to anything else. I almost never write on political topics and don’t really care where it’s hosted. I don’t even need many connections but do value some and the fact that Livejournal provides near zero ways to discover other blogs sometimes works more as a benefit than a drawback, because with the current state of things when majority of users migrated to Facebook or Instagram, there is not that much to read there. It’s not as engaging as Facebook, and that’s good. The other benefit of the service is that it’s a very good site to write long posts like this one and people actually do that and there is a ton of beautiful content there, if you’re lucky to find it of course.

My main problem with Livejournal always was that I didn’t really own the content in a sense of storage. The blog is mine, ok, but if web service goes down or account gets blocked I’ll lose all my ~1400 posts written to date without any good way to get them.

From time to time I tried to use other blog platforms or static site generators (like this site) and my real dream was to combine them and have sources in markdown locally, so that I can edit them in my favorite editor and still publish them on a platform that has at least some social flavor. Blog on Github + Disqus is not social for sure.

I had this idea for some time and somewhere around 2016 got really burned out by trying to ship yet another web service in my spare time in addition to those that I ship for my company. To recover I decided to go in the direction that absolutely excluded any interest except my own and didn’t have any money or fame in sight. You see now, Common Lisp and idea about better blogging at the time where blogging is already out of fashion is almost a perfect match! And this is where it all started.

Idea

What does perfect blogging experience look like? It’s when you write the text in your favorite editor, save it, push it to git and it’s published. And it’s when you can just grep your posts to remember something from the past or when you can do bulk updates to all the posts with system tools that you’re used to and get all changes published at once.

This was my thought process and this is what became a base of requirements. A client should be able to:

See new markdown files and publish them
See modified markdown files and submit changes
See deleted markdown files and delete them in the service
Bonus points if it all happens automagically on git commit/push

That’s a simple version of the client that implies that a folder with markdown files acts as a master database that gets replicated to a slave which in this case is web service. An obvious drawback here is that I lose any other ways to update content, be it via service web interface or it’s mobile app. So an even more perfect client should be able to:

Fetch any updates from the service
Transform them into markdown format so that all synced posts can be later edited in a usual manner.

One final thought was that since all editing happens in an ordinary editor, an obvious way to implement client is to make a cli tool for that. And if you check requirement you’ll spot that they resemble typical git operations a lot and it makes sense to mimic git whenever it makes sense.

Common Lisp nontrivialities

When you start working on a really big task, the only thing that can be said for sure is that there are a lot of unknowns. And if in a familiar language most of the unknowns lie in the business domain, in my case a pile of unknowns also waited to be found on the language and ecosystem side. Some of the things that I thought to be trivial and that were doable in many languages appeared to be not trivial at all in Common Lisp and that required a lot of time to grasp to get a full picture.

Let me go through some of the topics that appeared to be nontrivial.

Current working directory

This is one such case. It’s just one parameter, what can go wrong? Well, it’s two, at least in sbcl. It appears that there is a distinction between cwd of external commands run by uiop/run-program and internal lisp facilities. Let’s make a following experiment:

# preparation
$ mkdir -p ~/test_cwd/second_folder && cd ~/test_cwd
$ touch test.file
$ touch second_folder/nested.file
$ sbcl

and lisp session:

* (ql:quickload :uiop)
To load "uiop":
Load 1 ASDF system:
    uiop
; Loading "uiop"

(:UIOP)

* (probe-file "test.file")

#P"/home/can3p/test_cwd/test.file"

* (uiop/run-program:run-program  "ls"
    :input :interactive
    :output :string))

"second_folder
test.file
"
NIL
0

So far so good!

* (uiop/os::chdir "second_folder")

0
* (uiop/run-program:run-program  "ls" :input :interactive :output :string)

"nested.file
"
NIL
0
* (probe-file "nested.file")

NIL

Oops. Yes, probe-file expects something else.

* (setf *default-pathname-defaults* #P"second_folder")

#P"second_folder"
*  (probe-file "nested.file")

#P"/home/can3p/test_cwd/second_folder/nested.file"

And to double check.

* (setf *default-pathname-defaults* #P"/home/can3p/test_cwd/")

#P"/home/can3p/test_cwd/"
*  (probe-file "test.file")

#P"/home/can3p/test_cwd/test.file"
*  (uiop/run-program:run-program  "ls" :input :interactive :output :string)

"nested.file
"
NIL
0

And that trailing forward slash in pathname is important, don’t lose it. As you see, *default-pathname-defaults* and uiop/os::chdir are independent and influence different things and to keep it consistent for both standard library and external calls it’s necessary to call them both.

It’s worth saying that this distinction is of particular importance for repl based development, because when the image starts these two values are most probably aligned.

Another note is that the behaviour described belongs totally so sbcl, other implementations may behave differently.

Since I mentioned uiop/run-program, let’s see how I used it.

User input

User input is hard in a sense that it’s not only about Common Lisp versus the world it’s also about different operating systems, and it’ll be nice if all user input works in emacs repl too.

The simplest ever way exists for yes/no questions and it’s as simple as

(when (y-or-n-p "Do the thing?") (do-the-thing))

But let’s say that you want to read an actual input. There is a read-line function that can help. One complication is a prompt message that you may want to display. Simply formatting it to STDOUT will result in bad results in emacs in a sense that prompt text will be shown after actual input. To solve this I found that it’s necessary to force output on a used stream and now prompt function looks like this for me:

(defun prompt-read (x)
  (format *query-io* "~a: " x)
  (force-output *query-io*)
  (read-line *query-io*))

And that’s not all, because there is another possible user input which is passwords. Why is it unique? You probably will not want to enter it clear text. Unfortunately, I didn’t find a proper way of doing this magic inside of lisp and in the end the solution was to use stty to manipulate input visibility and run-program and internal bash read function to get user input:

(defun prompt-read-password (x)
  (format *query-io* "~a: " x)
  (force-output *query-io*)
  (let ((password (run-program "stty -echo; read val; stty echo; echo $val" :output :string :input :interactive)))
    (string-trim '(#\Newline #\Space) password)
    ))

Unfortunately this workaround doesn’t behave well in Emacs slime session.

Storing passwords

Next two are not Common Lisp specific but might be of interest for the curious ones. Since I read the password I wanted to store it to avoid asking for it again and again. However, I wasn’t fond of storing it in clear text and decided to explore system-specific solutions.

Mac os is perfect in this case because of built-in program security that allows to store and retrieve passwords. For Linux (Ubuntu in particular) there are no built-ins, however, after some search I found secret tools that provided similar functionality:

(defun get-password-cmd (login url)
  #-LINUX
  (format nil "security find-internet-password -a ~a -s ~a -w" login url)
  #+LINUX
  (format nil "secret-tool lookup '~a:login' ~a" url login))

#-LINUX
(defun set-password-cmd (login url password)
  (format nil "security add-internet-password -a ~a -s ~a -w '~A'" login url password))

#+LINUX
(defun set-password-cmd (login url)
  (format nil "secret-tool store --label='~a' '~a:login' '~a'" url url login))

Why is set-password-cwd signature different? secret-tool will read the password for you and security expects it as an input.

Packaging and running

Since the client is a cli tool, it’s really desired to compile the client system somehow. It’s not difficult by itself but I decided to place an additional restriction there and make it easy to run and install it both in a dev environment and as an easily installable package.

A task of running the system as a script is perfectly solved by roswell script and compilation is done with a buildapp. Since script cannot be used by buildapp to build a system and roswell and buildapp pass slightly different parameters to the entry point I ended up moving entry point logic into a separate package.

Buildapp itself works amazingly well in case lisp and quicklisp are set up on the computer, however, I wanted to make it really easy installable and attaching a binary file to Github release doesn’t sound user friendly for cli tool. The usual way to install a program on Mac OS nowadays is via Homebrew. A specific requirement for homebrew formulas is not to use any kind of additional package manager to build a program, hence quicklisp could not be used as is and I looked for a workaround.

The result was cl-brewer system that generates a Homebrew formula with urls pointing to all the dependencies so that formula can download them all and then use quicklisp and buildapp to make a binary without any additional downloads.

For Linux distributions I didn’t bother to create a package, however making a deb-package or a snap or whatever looks easy enough. It just waits for a hero that will do it.

A little note about arguments parsing. There are libraries for this task in the ecosystem but I ended up using none of them. I wanted to have a command based cli with some interactivity and the only things important in this case were first and maybe the second argument.

Editing

One last bit of user input! What I wanted to do is to allow to invoke a user editor (set by $EDITOR environment variable in case it’s necessary from the cli tool. That also appeared to be not so trivial, because actual implementation differs by Common Lisp implementation used.

Fortunately, I found magic-ed on Github and it magically did the right thing (again, not from inside of emacs, which is unfortunate). Quicklisp didn’t have it and I ended up including it in my code to simplify the build process.

Pre-commit hook

This one is not really Common Lisp specific, but it turned out to be pretty useful. What I really wanted to do is to use the client with git to have all my posts under version control. In this case the best ever flow looks like this:

$ cl-journal new post-slug
$ # thanks to magic-ed, an editor opens and you can write your beautiful post
$ git add . && git commit -m "another post"

After that new post file along with all the generated metadata should be included in the commit. A trick here is to include all changes in the pre-commit hook. Apparently, git allows that. Here is how current cl-journal pre-commit hook looks like:

#!/usr/bin/env bash
cl-journal push
git add posts.lisp

Of course, I didn’t figure everything out beforehand, but it took long enough to be worth sharing. Now let’s get to the client internals.

Perlotto - youtube support

2018-05-30T00:00:00+02:00

Ouch: This post was supposed to be published back in September ‘17, but I saved it in a future date. Here you go, anyway.

One more update about Perlotto app (last update is here). I decided to work on it again, because of annoyance I felt from using youtube in web browser. It’s just dumb to go from a decent players we all had to a web page that doesn’t work with multimedia keys to say at least.

Integration appeared to be quite easy and now it’s more or less trivial to extend perlotto to work with your favorite service (pull requests are welcome, btw).

The whole integration part is split into several things that need to be done to make service work:

How to understand that player page is loaded
How to understand that player is playing
How to do play/pause ot switch to the next or previous track
How to get information about current song.

The last one appeared to be the trickiest, and the reason is that the only way to get this data is to parse title and description, which are free form. For now I decided to go with simple spitting of the title string, but I’m totally sure, that it can be done hundred time better.

One part that I haven’t mention in the list is ads. I’m more or less fine with them, but it appears the without mblock origin youtube plays ad video after every second song and it was annoying to extend that I decided to cut it.

Electron provides a quite convenient api for this usecase, it’s possible to control and intercept requests on every stage of the lifecycle. The only thing that’s tricky is how to setup filters, because filter argument looks very misleading. After some trial and error I ended up with this code:

    const filter = {};
    function initFilters() {
        session.defaultSession.webRequest.onBeforeSendHeaders(filter, (details, callback) => {
            callback({cancel: checkFilters(details.url)});
        })
    }

Calling this function will force electron to pass every single request via the callback, where the actual filter lives:

    function checkFilters(url) {
        return ad_filters.reduce(function(acc, filter) {
            return acc || !!filter.exec(url);
        }, false);
    }

ad_filters is an array with regexes that are applied to the url, any match leads to the blocking of the request (full code there).

Current support is rudimentary and can certainly be improved. Some of the possible features are:

Smart track info discovery. With some more logic it’s possible to achieve much better results with decoding track, artist and album title.
Full albums handling. Putting full albums on youtube is really popular now, and most of the times you can see a full tracklist with timestamps in description or one of the comments. It would be a reall dealbreaker to enable perlotto to read this informaton and to integrate it with all the media keys and track information.

Outro - spotify

Another service that I decided to integrate was spotify. All the steps seemed to be pretty easy, except that I’ve found out that spotify web player doesn’t work in the perlotto due to abscence of widevine plugin.

Adding support required much more work - it should be downloaded separately and version of the plugin should match version of the chrome browser, which sounds like a lot of troubles without real benefit since I’m not a frequent spotify user.

Maybe I’ll do it some day, but if you’re interested, please don’t stop yourself from doing a pull request!

Building electron app - last.fm support

2017-06-25T00:00:00+02:00

Half a year ago I’ve built and Electron app to turn Google Music service into something behaving like a desktop app. The overal setup turned out to be exceptionally stable - there was no need for any change since then.

Last week I checked out my Last.FM profile and I though it’s quite a miss that I stopped collecting all the stats about the playback, mostly because there is a real discovery problem with a large music collection - you listen to small fraction of it and forget about older additions as soon as you switch to the new ones. And since you forget you don’t have an easy way to ask Google Music to play something that you liked in 2015. On the other hand Last.FM gives users decent analytics and some discovery tools which are quite handy for me as well because I don’t use Spotify.

So, last week I finally decided to spend time implementing it and there was a couple of issues I’ve spent a plenty of time on, that are worth talking about. So, here we go!

This is a general flow I wanted to implement:

User launches the app first time
There is a menu item “Connect to Last.FM”, user clicks on it
New window is opened with Last.FM authorisation page
After user confirms, window is closed, app starts scrobbling from the next track.
Menu item is changed to “Disconnect from Last.FM”. If user clicks it app deletes session information, menu gets back to the restored state. If not, user should get into authorized state after launching app next time.

Track information

First of all I decided to check whether it’s even possible to discover information about new song without dumb polling or hacking into compiled js. There were no obvious events flying around, but the dom looked well structured, so just two functions were needed to understand the current state of the player:

function isPlaying() {
    var el = document.querySelector("#player-bar-play-pause");
    return el && el.classList.contains("playing");
}

function trackInfo() {
    return {
        track: document.querySelector("#currently-playing-title").innerText,
        album: document.querySelector(".player-album").innerText,
        artist: document.querySelector(".player-artist").innerText
    };
}

I don’t do any error checking there at the moment, but it’s that simple. Now, that I can get the desired information, the solution is to store the information about current track and send an event outside whenever track is changed:

var currentTrack = {};

function tracksEqualp(one, two) {
    return one.track === two.track
        && one.album === two.album
        && one.artist === two.artist;
}

function analyzeTrackChange() {
    var playing = isPlaying();
        if (!playing) return;

    var info = trackInfo();

    if (!tracksEqualp(currentTrack, info)) {
        currentTrack = info;
        scheduleScobbleUpdate(info); // I'll come to that later
        ipcRenderer.send('player-song-change', info)
    }
}

Really simple. Now we need to figure out when should I call this function to understand wether new track started playing. Luckily we have Mutation events now and we can simply subscribe to changes of the event we’re interested in.

function runWhenTrackInfoChanges(func) {
    var target = document.querySelector('#playerSongInfo');
    target.focus();

    var observer = new MutationObserver(() => func());

    var config = {
        childList: true,
        subtree: true // see crbug.com/134322
    };

    observer.observe(target, config);
}

Events work amazingly good - I’ve tested it and event is fired only once in my case. So now the only thing remaining is to understand when we can actually start listening for changes. It’s obviously not DOMReady, and I reverted to polling there.

function playerLoadedp() {
    return !!document.querySelector("#player-bar-play-pause");
}

function runWhenLoaded(func) {
    if (playerLoadedp()) {
        func();
        return;
    }

    var timer = setInterval(function() {
        if (playerLoadedp()) {
            clearInterval(timer);
            func();
            return;
        }
    }, 200);

So, all necessary pieces are there, and what’s left is to plug them together:

runWhenLoaded(runWhenTrackInfoChanges.bind(null, analyzeTrackChange));

Now, that would be enough if we only could notify Last.FM right after track change, but scrobbling has some limitations - it api should be called later then 30 seconds after start of the playback + there are limitations for the track length. Now that I think of it, the best solution would be to fire the scrobble event on track change as well, but back then I decided to have a scheduling function for that:

var updateTimer;

function scheduleScobbleUpdate(info) {
    clearTimeout(updateTimer);

    info.timestamp = Math.floor((+new Date()) / 1000);
    updateTimer = setTimeout(function() {
        ipcRenderer.send('player-scrobble-time', info)
    }, 61000); // a bit bigger then minimal limit
}

Timestamp is necessary for the Last.FM api. Whenever user or app switches a song, timer is reset and scheduled again, so that I don’t fire events when it’s not needed.

Why did I decided to go with events? The reason is that they provide a good abstraction and decouple track notification from actual scrobbling calls.

Here is how things look like in main thread, after everything is implemented:

ipcMain.on('player-song-change', function(e, arg) {
    lastfm.nowPlaying(arg);
});

ipcMain.on('player-scrobble-time', function(e, arg) {
    lastfm.scrobble(arg);
});

lastfm.init();

In future it would be trivial to add notifications and other bloat there, but it’s not my future, consider forking if you really need it!

I follow a bottom-up approach when developing new features. That really helps - instead of trying to do everything at once I can slowly create building blocks that will make final solution trivial. Providing events was a first step (itself made from small building blocks). After that I decided to check if I can manipulate menu at all. It appears that it’s quite simple, just a bit unusual.

It’s not really possible to manipulate individual items of the menu, the only working way was to create a full menu and replace everything. If you think about it, it’s even better. First, template:

function getMenuTemplate(lastFMEnabled) {
    var template = [
        {
            label: app.getName(),
            submenu: [
                {role: 'toggledevtools'},
                {type: 'separator'},
                (lastFMEnabled ?
                    {label: 'Disconnect from Last.FM', click: function() { lastfm.disconnect(updateMenu) } }
                  : {label: 'Connect to Last.FM', click: function() { lastfm.authorize(updateMenu); } }),
                {type: 'separator'},
                {role: 'quit'}
            ]
        }
    ];

    return template;
}

Whenever menu has to be modified, we call updateMenu function that gathers everything necessary for proper render:

function updateMenu() {
    var lastFMEnabled = lastfm.isAuthorized();
    var menu = Menu.buildFromTemplate(getMenuTemplate(lastFMEnabled));
    Menu.setApplicationMenu(menu);
}

Now, the only thing remaining is actual authorization with Last.FM

Last.FM Authorization

Authorization logic has following requirements:

When user clicks on authorize link we should open authorization window
If user aborts the process / closes the window nothing should happen, user should be able to start again
If user grants permission, we need to close the window and update the interface.

After some research I came to a conclusion that the only viable flow is to use Last.FM authorization method that redirects user to the specified url on success. Local urls (file:///) did not work for me with any options, so I decided to go with local webserver.

var port = 4567;
var host = 'http://127.0.0.1';
var connect_url = () =>  host + ':' + port + '/auth';

exports.startServer = function(cb) {
    if (server) return;

    port++ // we don't want to fail if we did not stop previous server

    server = http.createServer(function (req, res) {
        var pathname = url.parse(req.url).pathname;
        if (pathname === '/auth') {
            var token = url.parse(req.url, true).query.token;
            exports.stopServer();
            cb(token);
        }

        res.writeHead(200, { 'Content-Type' : 'text/html' });
        res.end('');
    });

    server.listen(port);
}

exports.stopServer = function() {
    if (!server) return;

    server.close();
    server = null;
};

Since we only need to understand if interaction happened, simple callback is enough, connect_url will be used in other parts of the program. Please note the trick with the port - I decided to go for it since stopping the server doesn’t necessarily mean that connection is closed, so I decided to go with changing the port to be on the safe side. Frankly that’s not enough since chosen ports might be used, but I decided not to fight with that corner case for now. Also, functions are exported for testing purposes. And now, main part:

exports.authorize = function(cb) {
    if (lfmWindow) {
        lfmWindow.focus();
        return;
    }

    lfmWindow = new BrowserWindow({
        width: 600,
        height: 600,
    })

    exports.startServer(function(token) {
        exports.stopServer();
        lfmWindow.destroy();
        lfmWindow = null;

        lfm.authenticate(token, function (err, session) {
            if (err) { throw err; }
            sessionData = session;
            settings.set('lastfm', session);
            cb();
        });
    });

    lfmWindow.on('close', function(e){
        lfmWindow = null;
        exports.stopServer();
    });

    var url = lfm.getAuthenticationUrl({
        cb: connect_url()
    });

    lfmWindow.loadURL(url);
}

exports.disconnect = function(cb) {
    settings.sef('lastfm', null);
    sessionData = null;
    cb();
}

Again, there is nothing we want to understand except whether authorization succeeded, so I go with a simple callback and had all handling logic inside. Settings are saved with synchronous library, since I did not want to deal with it for the sake of simplicity.

After authorization part is done, the only thing needed is a proper initialization:

exports.init = function() {
    sessionData = settings.get('lastfm');

    if (sessionData) {
        lfm.setSessionCredentials(sessionData.username, sessionData.key);
    }

    return null;
}

exports.isAuthorized = function() {
    return !!sessionData;
}

To complete the module I decided to add functions for scrobblying and updating the service about currently playing song. To avoid complicating interface in the main module they both fall back to noop if Last.FM integration is not set up.

After this point I’m ready to say that Perlotto reached full functionality state, and hopefully I won’t need to update it for another year or so. :)

Finally, I hope some of the tricks described in the post will save few hours to devs struggling with them.

Common lisp - good parts

2017-05-12T00:00:00+02:00

Since last summer I decided to invest some time into learning common lisp and wasn’t disappointed. Certainly, the language shows it’s age and there are some issues with documentation and libraries, but the benefits are so great that it’s hard to return to usual languages of choice.

A week ago we had a first Lisp and Scheme meetup in Amsterdam (please, join the next one!) and I’ve prepared a little talk to discuss what features I find especially affordable and powerful.

Since I’m not a perfect speaker by any means, here is a text version.

Intro

Tech culture is really spoiled by hacker news syndrome. If some corp © has a budget big enough to promote the language, we’ll see it all over the place, and of course, everything starting from cli parser and editor and ending with distributed systems will be implemented there. If they scream loud enough, it’s really easy to assume that these concepts or methods were invented there.

That leads us to a skewed perception of the world. We always think that we’re moving from some sort of dark ages towards something looking like bright future. It could be so but it’s not necessarily so.

Having no formal CS education certainly does not help. I started my career as a developer working with Pascal and Delphi, then moved on to the web development path where I mostly lived inside of a page request with languages like PHP or web page lifetime with Javascript. Although I did progress with understanding the whole stack from bottom to the top I was exposed to one specific way of doing things which is dynamic languages and it bothered me since web development is detached from system programming and associated languages. At some point I encountered clojure and it was no less than a turning point in my life. The experience was so different that to this point I can’t imagine spending my free time on something less powerful.

The core of the experience was of course in the interactive development which makes development so much faster, I’ll talk about it later.

When you have a language that powerful you want to write basically everything with it and having the jvm as a backend meant slower startup times which is not acceptable for cli programs. For sure there are hacks to fix that, but I wanted a cleaner solution and of course every flamewar around clojure had some sort of comparison to common lisp. So in the end I decided to go back to the roots and to try it out and learn the language.

So, common lisp. Some of it’s concepts may look surprising but were not uncommon in the early nineties. I’m not a Common Lisp expert, so the rest of the post is not an authoritative source, but more my perception of the language at the time of writing. If you see anything clearly wrong, please tell me.

I’ve taken three different aspects which are notable for me one section for each. Here we go.

Stability

A small section for the big feature. Stability of Common Lisp as a language and platform is remarkable. It’s specification was in the works for ten years and hasn’t been changed since original publication. There are constant debates around this, every year or two one more attempt to modernize language pops up. For sure they target some problematic areas but problems are not so severe for these projects to get traction.

From the other side specification is really complete, so you can see a defined behavior for any corner case (even if it’s specified as an implementation defined - these places usually talk about the internals of the language). There are around 10 different compilers that are actively developed and provide different sets of features and most of them have been in development for years so you can imagine the level of robustness there.

I would say that sbcl is more or less a default choice at the moment, and there are differences between implementations as there are compiler specific packages and apis, so it’s not all roses.

Code that was written fifteen or twenty years ago can run on any modern compiler. How cool is that?

Interactive development

Interactive development is one of the major features of the common lisp and a cornerstone of the language design. Everything was made in a way to allow this method of development.

The easiest way to show this is to outline how development happens in common lisp and in classic languages.

Most programming environments do not provide too many tools to understand and change the state of the running process, I’m not even talking about actual development there. In some the only way is printing data into STDERR, in others you can use a debugger to set a breakpoint and look around. Frontend developers are luckiest in this case - web page is really easy to introspect by definition and there are ways to do bits of interactive development there (just look at all the excitement about css hotloading) and it’s also a place where it’s easy to feel the difference. Once you have a dynamic page and build pipeline you immediately lose in productivity and it requires more and more efforts to compensate. In other environments you face a necessity of building program interface from day one - write cli interface, make assumptions about the structure of data and prepare some stubs, understand how this data gets into the program and back from it. Depending on requirements it can take few hours or few days to get this fully working, and that’s the time you could spend on solving the main problem.

Anyway, after finishing with that you get into a usual cycle of development. You make a change, change all the data storage etc. to accommodate for new data structures, compile and restart server / run tests, where you’ll load all stubs and hopefully get the desired result. If not you penetrate your program with all sorts of print statements just to understand what happens.

Interactive development in common lisp is different in sense that you start developing by starting repl-process and you can already start solving problem and write necessary business logic, all the stubs can be generated programmatically and put into the variables as you go, No cli interface is necessary. While being attached to the process it’s possible to access any package, inspect any variable, change any data structure and define or change any new function. That means that change cycle is reduced to change code -> recompile function and all proper interface is moved to a later stage.

All this does not deny testing, it’s just that you do not rely only on it and can have much quicker troubleshooting cycle. Moreover, in many cases you can live without a proper external interface for a long time or not need it at all if your task is only to calculate something or prove the idea and you won’t lose any flexibility because you’re calling your code directly by passing language data structures. Btw, this is another advantage which is everywhere in common lisp - in many cases you don’t need to parse strings or invent serialization formats you can easily use language data structures.

That’s actually how any lisp program works - lisp process starts, loads and compiles some lisp code and runs some function on it. Why it works like this is because CL is an image based language - during the interactive development session developer shapes the program to a particular state where he can dump the image into a binary file and this is a default way of compiling programs in common lisp.

(ql:quickload 'trivial-dump-core 'your-awesome-project)
(trivial-dump-core:save-executable
  "awesome_project"
  #'your-awesome-project:main)

What happens when you dump image? All the environment is dumped there -> you have a compiler etc when you run it again Startup time of the image is really fast - ~10-30ms Dumping state means saving all the memory state like variables and that means that theoretically you only need to dump image of problematic environment to get all the information Dumped image has no dependencies, so it can be safely deployed as one file.

Extensibility

An important point is that the word “Common” in Common lisp means that the language was created as a response to the existence of many slightly or very different lisp dialects that had to be united. On one hand, it resulted in some duplication (like structs and classes), on the other hand, extensibility is a cornerstone there are ways to extend lots of different aspects of the language or program.

Assignment

What can be simpler than plain assignment? Really, nothing to see there. We can assign to variable, or maybe have a setter on the object.

var variable = "value";
myObj.variable = "value";

In common lisp the operation is generalized to setting the result of form to a place.

(setf <place> <form>)

While <form> is easy, <place> is much more interesting, because variable is only one of it’s possible meanings.

(setf var "value") 

(setf (aref var 2) "value") 

(setf (some-field obj) "value") 

(setf (my-storage :key) "value")

You can define a function call as a setf, e.g. (aref arr 0), or basically any other form via the macro, so that you can have mirrored get/set interface in almost any imaginable case and it will work consistently via setf. For example all the getters for the object fields are defined with setf extensions.

Generic functions

Generic functions are one of the fundamental parts of the Common Lisp and they are fantastic in a way that although they are dispatched against types they are totally decoupled from them - all of them, not just the first one. Common Lisp automatically orders implementations against declared types and chooses the most specific implementation for every call. As a consequence new functionality can be easily implemented against any existing class hierarchy, even defined in the other library.

For example we work with some kind of DOM tree and want to print it in our specific way. We can define a generic function, give default implementation for it and add any number of specific implementations for certain kinds of elements.

(defgeneric print-xml (stream element &optional indent))

(defmethod print-xml (stream (element <element>) &optional (indent 0)) …)

(defmethod print-xml (stream (element <dict-article>) &optional (indent 0)) …)

And that means that you don’t have to make a wrapper on such classes or do inheritance just to add a method, you can just implement business logic against it.

Another advantage is that with generics we get another way to group functionality - around functionality and not around objects.

As if that wasn’t enough generic functions come with a complete support of aspect-oriented programming with :before, :after and :around modifiers. That adds yet another direction of extension and may libraries and common lisp as a language define generic functions as external api so that clients can add hooks in to particular points in the program without passing lambdas around.

Suppose there is a database of posts that exposes delete-post generic function. We can remove all the I/O operations from this library and plug it externally.

(defmethod delete-post :after ((db <db>) (post <post>))
  (save-posts))

By defining a generic function library exposes not only one but multiple extensions points which can be used without touching the library code.

When generic function is defined developer also defines a way how dispatching works. By default the most specific method runs, but you can use several others like sum, aggregate or even define your own.

CLOS

CLOS is an object system for common lisp that supports many features that make it very advanced.

Class in common lisp is a bag of slots. During definition developer defines properties of every slot, so that it’s possible to define how slots are read or initialized.

Since functions are detached from objects, it makes it easier to have multiple inheritance and CL supports that. Every getter on the slot is defined as a generic function with implementation specific to this class but nothing prevents to redefine getter/setter.

One of use cases for multiple inheritance is when you need to use two independent concepts. Let’s look at the following example that I stole from the “Art of Metaobject Protocol” book.

(defclass <rectangle> ()
  ((height :initform 0.0 :initarg :height :reader height)
  (width :initform 0.0 :initarg :width :accessor width)))

(defclass <color-mixin> ()
  ((cyan :initform 0.0 :initarg :cyan)
   (magenta :initform 0.0 :initarg :magenta)
   (yellow :initform 0.0 :initarg :yellow)))

(defclass <color-rectangle> (<color-mixin> <rectangle>) ())

(defgeneric paint (x))

We have a generic function paint and a <color-rectangle> class that inherits from <color-mixin> and <rectangle>. We can implement function itself against <rectangle> class and it will, well, draw a rectangle. After that we can implement :before method against <color-mixin> that will setup brush properties and maybe :after method to reset them. By doing that we decouple two separate concerns and it’s trivial to implement any other shapes with color support.

CL went one step further and made all the definitions in common lisp as instances of standard objects, which means that e.g. class definition is an instance of the class “class definition” and can be extended. With that CL provides multiple generic functions that are dispatched against standard objects and it’s possible to create a subclass of standard objects and change the behavior of the particular objects. As a result CLOS covers not only one specific way to implement an object system (although default way is provided) but a wide spectrum of possible implementations.

Macros

To get the idea about possible use cases for macros let’s consider the following example in some javascript like language.

class Widget {
    constructor(height, width) {
        this.height = height;
        this.width = width;
    },
    run() { /* useful stuff */ }
}
API.registerWidget("Widget", Widget)

Is there any code duplication there? In javascript you’ll say that there isn’t since all the lines are the necessary boilerplate to make things work. Let’s imagine a perfect world and choose the best syntax for this case. Wouldn’t it be nice to write something like this?

defWidget Widget(height, width) {
        /* useful stuff */
}

Here is a possible implementation in case of Common Lisp.

(defmacro define-widget (name params &rest body)
  `(progn
     (defclass ,name () ,(expand-params params))
     (defmethod run ((widget ,name)) ,@body)
     (register-widget ,(symbol-name name) ,name)))

Easy.

Macros provide a way to fight with code duplication and inefficient computation on the whole new level. Since macros are expanded and evaluated on the compile level, we get two advantages: we can offload intense computations on the compilation step if possible of course and code duplication can be chased now not only on the logic level but really on the syntax level.

In addition to that CL provides a way to define reader macros so that you can tune your program code to your liking. The big difference between lisp macro system and say c macros is that lisp macros work with a source code as a code data structure that is easy to manipulate with all the power language provides and not as a simple substitute.

Conclusion

I hope I’ve made a showcase for CL and you’re interested to try it out if you haven’t done it before. It’s not all roses and I have to give a word of warning there - the situation with documentation is mediocre, even some popular libraries can have problems there, there is plenty of outdated code or broken home pages. The language itself has duplication in some parts like mapping functions or binding variables and lacks some essentials like complete library to work with strings, community is much smaller than in any hype language.

On the other side there are libraries to compensate shortcomings of standard library and many interesting projects in different areas, some of them are really brilliant (see Mezzano as an example).

If you want to read about language in general, the best book ever written on CL on my opinion is Peter Seibel’s “Practical Common Lisp”. It’s an invaluable gift to humanity and what’s more it is freely available online. Another piece of art on the topic is “The Art of Metaobject Protocol” which is very interesting to check out to understand decisions behind the CLOS design. First part of the book is dedicated to the implementation of simplified CLOS and is amazing. And as I’ve told in “Stability” chapter, the fact that the book is from 1991 doesn’t mean it’s outdated :) There are other books like “On Lisp” from Paul Graham, but they can be read on much later stage to get to the new levels of enlightenment.

Quickdocs website is a place to go if you want to find some library.

If you want to get help #lisp and #lispgames on freenode.net have the highest concentration of lisp devs alive, lisp reddit is worth checking.

Have fun!

Building electron app

2016-09-13T00:00:00+02:00

I used Radiant player for a long time and didn’t have any complaints till the moment most of Google Music interface stopped working well with the app. I checked bugtracker multiple times and no one raised similar issues. Feeling myself weak in Objective C land I didn’t have any desire to go and fix things myself and, as a consequence found myself opening GM app in the browser.

For some reason I really don’t like having apps openned in the background tabs. First of all it makes restarting browser a pain, second multimedia keys are not supported. So, after a while I cloned electron quickstart repository and started launching google music there and few days ago finally found the time to make a complete app - Perlotto. It’s called like this just because this word sounds amazing and I wanted to name something with it.

Overall setup took less time than I expected although I had solve few common issues, and solutions will be bellow.

Packaging

Somce it’s a real app it should be packaged as a real app, I have no desire to run npm start every morning to start my music player. The task is really simple with electron-builder, although it require a bit more effort to get app packaged automatically, since why would anyone want to package app manuall all the time, right?

Building part is explained in electron builder readme.

Here is publishing part (I care only about Mac OS X at the moment). Travis CI can be used to automate packaging and publish attefact on github releases page.

My .travis.yml

osx_image: xcode7.1
os:
- osx
language: node_js
node_js:
- 6.1.0
script: npm run dist
cache:
  directories:
    - node_modules
    - $HOME/.electron
    - "test/fixtures/app-executable-deps/app/node_modules"
deploy:
  provider: releases
  skip_cleanup: true
  file_glob: true
  file: dist/mac/Perlotto-*.dmg
  api_key: $GH_TOKEN
  on:
    tags: true
    repo: can3p/perlotto

Few things there:

Specifying osx_image alone didn’t helpe to make travis run build on OS X, I had to add os section.
I specified version 6.1.0 for node, however something like 5.8.0 was used. Doesn’t really matter for me.
script is the same command that is used to build package locally
cache section is not necessary but helps to speed up process
Builder produces image with name like Perlotto-0.1.dmg, so I used file_glob to workaround specifying precise filename
api_key is a must however I did not want to put key on github, so I ended up specifying env variable and passing it there (it’s possible to specify env vars on settings section for the project)
on section specifies that I want to make build for tags only, however travis still makes builds for every commit. electron-builder is smart though and publishes artifacts only if it’s a real release
electron builder takes version from package.json file, but github cares about tags only, so I have to keep them sync myself.

I used travis cli tool to generate some parts of the file and stole others from the internet, however it’s absolutely possible to do everything without the tool via web interface.

So, this is how release process looks now:

Update package.json version and push
Go to github releases page and publish new version (same as in package json). One thing to keep in mind that electron-builder wants versions to be like x.y.z and tags like vx.y.z, an exception is thrown in other cases.

Closing the window

Default behaviour of electron quick start app is to exit application when user clicks close button. I’m not aware of any serious player that behaves like this hence this misbehaviour has to be fixed.

First thought is of course to prevent this on window close event, but the result is that it’s not possible to close app in any way other process manager.

So, trick there is to raise flag on app ‘before-quit’ event and check it on window ‘close’ event:

let forceQuiteApp;

function createWindow () {
    // Create the browser window.
    mainWindow = new BrowserWindow({width: 800, height: 600})
    mainWindow.maximize();

    // and load the index.html of the app.
    mainWindow.loadURL('file://' + __dirname + '/index.html');

    mainWindow.on('close', function(e){
        if (!forceQuiteApp) {
            e.preventDefault();
            mainWindow.hide();
        }
    });

    })
}

app.on('ready', createWindow)
app.on('before-quit', () => forceQuiteApp = true);

I run maximize method right after the start because I don’t want to worry about saving window dimensions anywhere

Media keys

This one is simple, the only moment is that keys should be bound with globalShortcut object available from electron module on app thread and actual dom manipulation happens in content thread. Here is relevant app code part:

function createWindow () {
    // ... init part ...

    globalShortcut.register('MediaPlayPause', function(){
        mainWindow.webContents.send('play-control', 'play-pause');
    }) || console.log('MediaPlayPause binding failed');

    globalShortcut.register('MediaPreviousTrack', function(){
        mainWindow.webContents.send('play-control', 'rewind');
    }) || console.log('MediaPreviousTrack binding failed');

    globalShortcut.register('MediaNextTrack', function(){
        mainWindow.webContents.send('play-control', 'forward');
    }) || console.log('MediaNextTrack binding failed');
}

We use event api to transfer knowledge from app to thread, here is content part:

var ipc = require('electron').ipcRenderer;

ipc.on('play-control', function(event, command){
  var webView = document.querySelector('webview#gpm-player');
  webView.executeJavaScript("document.querySelector('#player-bar-" + command + "').click()");
});

Since script has to be embedded into content, local file is needed, hence here is index.html

<!DOCTYPE html>
<html>
  <head>
    <title>Perlotto</title>
    <style>
        * {
            margin: 0;
            padding: 0;
        }
    </style>
  </head>
  <body style="overflow: hidden">
    <webview id="gpm-player" src="https://play.google.com/music/" style="height:100%;width:100%;position:absolute;"></webview>
    <script src="./content.js"></script>
  </body>
</html>

Margins had to be reset, otherwise webview is rendered with some margins.

Easy.

Conclusion

It’s funny, but this post contains 99% of project’s code. I treat project feature complete since I don’t care about notifications, tray icons and any other garbage that is useful only for distractions.

Electron ecosystem is surprisingly feature complete and stable for me, thanks guys!

On Common lisp

2016-07-02T00:00:00+02:00

Though I have some experience with clojure which is lisp I always had a feeling that this experience is incomplete without trying other dialects, especially common lisp. This post describes my experience on starting with this language.

I’ve read a couple of books, namely Practical Common Lisp, Land of lisp, Full stack lisp and Let over lambda and they all are beautiful. Macros part is especially mind blowing because you simply don’t think that way in most other languages, and I still not used to them in a sense that I don’t write my own.

After reading them I decided to give it a try and write something. The result is my shiny new incomplete cl-journal client for Livejournal.com webservice. I set the following requirements for the client:

It should work with command-line
Password should be stored in Mac OS Keychain or any other system level application, not locally
The idea was that it should work as a pre-commit hook to replicate experience of static blog engines as much as possible.

While I found a solution for all these points, I was left with the feeling of very fragmented toolchain and ecosystem. If we compare it to clojure world, all project management and build tassk are done either with Leiningen or with boot, and clojuredocs.org is mostly enough to get all standard library documentation and finally most libraries have decent looking documentation pages. Every package has well-defined project file that contains configuration, dependency information etc.

It’s not like this in common lisp world (according to my limited knowledge). Dependency management is done with the help of Quicklisp, project scaffolding is done with Quickproject. Building is a whole different story since common lisp doesn’t build standalone binaries in usual sense, here the way is to dump current state of lisp process (including compiler and documentation) and define an entry point and run it whenever this image is executed. There are several projects that aim to simplify building and running applications, most notably Rosewell and Buildapp. Rosewell looks really great, my problem with it was that I haven’t found a way to install scripts from local repository, so I had to do manual symlinks here and there to make it work.

Since there is no central place for the documentation, I mainly used Practical Common lisp book, common lisp cookbook and quickdocs website.

One of other unsolved issued for me was password prompt. I didn’t find a proper nonhackish way to do it with common lisp itself, so I had to run external command for exactly this purpose.

For running commands I had to use uiop library, especially uiop/run-program and uiop/os packages. Package there is more like a namespace in clojure and clojure package is a system in common lisp terms. uiop/os namespace was especially useful because standard library doesn’t provide any functions to control current working directory.

Most of things described can be blockers only with first steps in the language and I believe that once all they fit in head programming in common lisp gets not harder than in any other language but with additional cool features.

The coolest one is of course an interactive development. Basically I’ve started repl just once or twice during development, all the features were written and tested in it and that had huge consequences for development process. If we take any usual language, you work there by continuously running the whole program or firing one-liners to get result from desired function. The drawback of former is that if initialization step is costly, your development speed can suffer from it a lot. The drawback of latter is that most of function cannot work in isolation - they need specific datastructures initialization of global data object etc, and if it’s not necessary, developer still has to take care of proper printing of the results. So, sometimes it’s just to much of work to run things in this way, so people revert to the first way. First way also suffers from the fact that instead of implementing business logic developer spends a lot of time on boilerplate code for passing parameters, entry functions etc. Interactive development eliminates all that hassle.

My client had to load local database with information of already published files before doing anything. Instead of loading it on every change, I did it once and then simply tested new functions that I wrote with it. And since I could do it, I did not implement boilerplate code till the very last stage when I already had working business logic in place.

Interactive approach is very different also in sence that bottom-up approach feels really natural there. E.g. In order to create a post I need to read a file. How do I do that? (= new function). After I read, how do I parse it (= new function). After I parse it ho do I understand that it’s not a draft (= new function predicate) etc etc. So I continued asking questions and putting my answers as functions there till I get to very top where I wrote simple function that really created the post.

As a result most functions are really small and easy to reason about. You can go bottom-up in other programming languages, but since you either have to write a lot of boilerplate code to get feedback or right a lot of tests which really slow you down when you’re just trying to find a way to solve the problem.

If you ever did any clojure programming you know that it provides similar level of experience. Where common lisp wins is a startup time for a built program (instant) and human readable stacktraces that are not polluted with java classes etc and excellent restart functionality instead of simple exceptions. Restarts are super cool, because they don’t just throw warnings, they also provide options to recover and sometimes error can be just worked around with this.

As a resume - language is definitely worth looking at and using for real things. I have very limited exposure to the language to talk about good parts and bad parts, but a scattered and some times very brief documentation adds to a challenge of learning the language. Many libraries that are widely used have there latest release a few years in the past, and documentation is not always nice (e.g. I had to read the code to understand how to use uiop packages properly. The whole ecosystem has a few active members which means that not much happens in a public space.

However there is a suprisingly big number of common lisp implementations that are really high-performant and battle tested. I do think that if language gets a proponent like Yehuda Katz, who will put effort into popularisation of the language and systematic toolchain enhancements, common lisp can get a great level of adoption.

Perl is nice

2016-04-25T00:00:00+02:00

There are many trendy things in wht wild and perl is definitely not one of them. Many blame it to be write-only, and it really has some ugly parts, there are things where it definitely shines. There are lots of parts that make data juggling easy.

Lists

My personal favourite is a list concept of the language. By default all lists are squashed to one level and every function receives nothing but a list of arguments. All hashes are lists with odd values used as keys and even values used as values. And in the end, => operator used in hash definition is absolutely equal to comma operator.

If you want to get a nested structure you should use references.

This is absolutely awesome concept that makes so many operations compact and natural that other languages like javascript should be really jelaous about it, because it allows to get things like destructuring trivially and makes conversions between lists and hash-map natural.

First of all, there is no nesting:

(1, 2, 3)


# equivalent
(1, (2, 3))

It’s not only about raw lists, it’s about lists as a type, so all this is the same:

my @arr = (2, 3);

(1, @arr)

sub func { return (2, 3); }

(1, func() )

Yeah, any expressions can do

(1, ( $all_good ? (2, 3) : () ))

Since hash map is initialized with the same lists (it’s stored as a hash map structure actually), all tricks still apply, the only restriction is that number of list items is even.

my %hash = ( some_key => some_value, get_options() );

All this allows to have idiomatic way to optionaly set key to hash:

my %options = (
        required_option => 1,
        ( $condition_holds ?  (optional_key => 1) : ())
        );

Or use map to transform list to a hash.

my %hash = map { $_ => 1 } @keys;

What happens is that we take a list and use map on it, where block is executed with every element and returns a list ($item_of_list, 1). In the end all returned lists are squashed to a single one which is used to initialize a hash. As a result we have a hash of all members of a list as keys and ones as values.

It’s particularly cool comparing to javascript where there is no way to achieve the same level of elegance:

// We can do like this:

var hash = {};
list.forEach(function(item) { hash[item] = 1; });

// or like this:

var has = list.reduce(function(acc, value) {
    acc[value] = 1;
    return acc;
    }, {});

For optional keys it’s the same level of awkwardness:

var options = {};

if (condition_holds) { options[key] = 1 };

While not a hard thing to do, it pollutes code with unnecessary variable declarations or loops that do very simple things and distract attention for the main logic.

Now, all this applies to any function calls, because what we pass is just a list of arguments:

func(1, 2, 3)

# the same

my @a = (2, 3);
func (1, @a);

Inside function we can parse arguments in any way we want:

sub func {
    my ($arg, @list) = @_; # one argument and the rest as a list

    # we can call this like func(1, key => value) for example
    my ($arg, %hash) = @_; # one argument and the rest as a hash
}

As you see, we get simple form of destructuring easily with a natural uniform syntax. Perl obsession with list went even further. E.g. you can set multiple keys of the hash at the same time:

@hash{qw(key1 key2)} = (1, 2);

As you see, this pattern allows to cut out a lot of boilerplate code and provides really effective tools for data structure manipulation.

Default variables

This is one of the features the brought perl a fame of write-only language. The reason is that It’s used for most language constructs and can make the code really dense and almost unpenetrable to anyone not so similar with perl.

I’m not aware of any other language that has default variables but with a wise use code actually looks cleaner. There are two main variables that are used as default ones - $_ and @_, which are scalar value (number, string, reference) and list.

All function arguments in @_:

sub func {
    my ($arg1, $arg2) = @_;
}

All loops and operators like map or grep make $_ point to current item of a list

for (@arr) {
    print $_;
}

my @doubles = map { $_ * 2 } @arr;

If you write this type of code in any language that does not have this feature you’ll most probably end up with lots of variables named as item or something alike.

What’s more some functions and operators work with $_ if no explicit argument was provided, and here is the place when unfamiliar people can get really confused.

for (@lines) {
    chomp;
    s/abc/def/g;
}

# the same
for my $line (@lines) {
    chomp $line;
    $line =~ s/abc/def/g;
}

Looking at this example you may see that perl really doesn’t mind mutating existing variables. This again can save some keystrokes but makes it even easier to shoot yourself in a foot.

These are two features unique to perl as a language that allow to do lot’s of tricks that make code dense and expressive at the cost of additional context that any developer should be aware of to successfully read the code. I don’t think that it’s a bad thing by itself but most language developers chose different path to provide everything explicitly so decrease learning curve.

Big thanks to Oleg Komarov for reviewing this post.

Free data

2016-03-23T00:00:00+01:00

Data rules the world, right? Vast amount of information can be used not only to extracts some insights from it, it can be a weapon or a timebomb depending on the circumstances. Bad news for ordinary people is that they rarely have access or own information, they mostly ask for it. What if one refuses to answer?

Some years ago it was possible to buy it at least, but in the age of ubiquitous web services everything lives somewhere else and we have no way to control it. Services that we use everyday belong to very few major players, and it’s hard to imagine a scale of the disaster if google goes offline or is forced to stop providing services to the users of the country that doesn’t fit to one’s understanding of real democracy.

A lot of web sites heavily rely on crowd sourced data, but the only way users can access it is the way provided by this services. Is it fare?

I’d say not. I firmly believe that future innovations heavily rely on free access to different kind of information.

If you remember there was a period in mid 2000s, so called Web 2.0 where the trend was to create open APIs and give away as much information as possible to enable interconnection of the web. After some time this trend turned into decline, but we really need to get it back.

Every service that takes data from users should make it possible to get it back. If the data is anonymous then it should be possible to get it all. Just have a look on the amount of applications that use wikipedia data as a source. Another example is OpenStreetMap project.

Knowledge and Navigation is two pieces, communication is the third one. We really nead open dictionary and translation engine the will prevent dependency on single player and allow everyone to benefit.

Two more field that come into mind are history and repairs. Talking about the latter, how many times anyone got into situation the the devices he or she had had only one detail broken and there was no way to fix it because it was no longer supported/produced? It shouldn’t be that way, especially with emerging 3d-printing technologies. We need and open service that will contain models for parts of the devices that could be printed. I can imagine the future where every such repair is just a matter of downloading proper model and sending it to a printer.

And at last, what can we do with the history? The problem with it is that due to amount of information and globalization it’s way to easy to hide important changes behind silly news. How nice it would be to have an open database that will accumulate information about all events happening, so that it will be possible to make an analysis and separate real news from the garbage and see real connections between events that would otherwise go unnoticed.

Browser will fix this

2016-01-28T00:00:00+01:00

After working many years in the frontend development area I really got used to technology which led to some kind of blindness to the errors that can happen, I don’t see certain kind of errors anymore just because I got the habit of doing things certain way and it doesnt (usually) get me into trouble.

When you start writing markup you’re all about semantics and compact code. Isn’t it beautiful?

<body>
    <header>
        <h1>My blog</h1>
    </header>
    <section>
        <header>
            <h2>Great post!</h2>
        </header>
        <article>My awesome content</article>
    </section>
...

</body>

Yeah, it is. And maybe it even makes sence if you’re working on the site alone, or you’re planning to ship it to the client and run away because this kind of markup implies selectors like section header h2 {}. And while it looks nice and concise, things start breaking after you develop more than one page with shared styles and a product owner who always wants to see this block a little bit different.

The bigger project grows the more effort is required just to keep semantics on a proper level, which of course affects the velocity. After few more years you start to understand that the only difference about html tags is that some of them are inline and others are block-level. No, actually there are just elements and there is css that can make anything you like. If I were to write the same markup today, most probably I would come up with something like this:

<body>
    <div class="header">
        <h1 class="header__title">My blog</h1>
    </div>
    <div class="post">
        <div class="post-head">
            <h2 class="post-head__title">Great post!</h2>
        </div>
        <div class="post-content">My awesome content</div>
    </div>
...

</body>

Unique classnames, selectors with the lowest specificity possible. Doesn’t look beautiful, but I know for sure, that I won’t get into troubles with selector clashes, my css won’t consist on 50% of !important rules and since span and div element don’t get any default styles from the browser they usually have an overwhelming majority on the page.

The second benefit is that I can always move blocks around without breaking too much stuff, nest blocks into each other etc. Nesting part is a tricky one, because browser doesn’t really think that elements are the same. Here is the code:

<div class="content">
    <p class="content-head">
    We start good
        <div class="reusable">Block</div>
    Ouch
    </p>
</div>

Apart from the fact that div looks odd inside of the span, what can go wrong? Here is what browser thinks (codepen example):

Yeah, semantics.

The other nesting pitfall that is possible to get into is with forms(codepen):

<form id="form1" action="/post">
    <input name="test">
    <form id="i-dont-know-how-i-got there" action="/get">
        <input name="login">
        <input name="password">
    </form>
</form>

The last one is my personal favourite, because the misbehaviour can be introduced a loong time before you will introduce the changes that actually trigger the bug:

<div>
<form id="form1" action="/post">
    <input name="test">
    <input name="login">
    </div>
    <input id="password" name="password">
    <span id="spacer"></span>
</form>
</div>

Mind the closing </div> in the middle of the form. It can really happen if you have a huge project with lots of nested templates. Here is inspector view:

You see, last input field falls out of the form. The browser magic is in the fact that it still belongs to form (check another codepen)!

The bug becomes visible if you insert another input dynamically, like this:

document.querySelector('#spacer').innerHTML = '<input id="out" name="out">';

This input won’t belong to the form at all (codepen).

If you think about it, all this absolutely makes sence, because javascript works with already fixed DOM tree and the element is inserted outside of the form, but it can take quite some time to spot this in a huge codebase.

Forgot to mension, ids are here only for test purposes and ideally should be used only in situations where html spec wants them (like <label for="" />).

Advent of code quiz reflections

2016-01-05T00:00:00+01:00

This December a big chunk of my free time was spent on solving quize from Advent of code challenge. While I wasn’t anywhere amongst the leaders it was a pure joy to see how clojure helps to gradually build solutions without constant switching between the editor and console.

If the only thing you have is a hammer then everything looks like a nail and majority of tasks ended as some sort of list generation which is somehow reduced to the to the answer value. If you look at the task from this point of view you will discover that it really helps to avoid most of side effects or temporary states which can be a big source of bugs.

If you think about everything as list it becomes immediately evident how rich and cool core library is. Apart from it I used only clojure.string, clojure.set and math.combinatorics namespaces.

The main disadvantage for me was that with certain number of map/reduce/filter functions in one expression code becomes much harder to read for anyone who is not very familiar with them and sometimes it’s evident that in some cases types can really help because it’s hard to say how structure looks like at given point of transformation. May record is 9 map/reduce calls in one function.

The other pain point is emacs cider and changing project dependencies. Maybe I don’t read documentation correctly but the only good way to load new dependencies is to restart repl and to do that I had to kill it. Sad news is that after that in most cases I ended up with restarting emacs which meant losing undo history and repl logs.

Ok, back to interesting stuff. I want to share a few incredibly useful functions from the core library that I discovered while doing the challenge. Although simple sometimes they drastically simplify the code.

reductions

Used in day 1

This function takes the same arguments as reduce but instead of final value returns a sequence of all intermideary reduction states. It’s useful if every reduction step has it’s own meaning, e.g. distinct state that you want to analyze. Or if you have an infinite sequence you can use reductions to count the steps to get to the desired state.

every-pred

Used in day 5 and day 11.

Takes any number of functions and returns new function that will return true only if all the functions return true with arguments passed. Very useful to write conditions in compact way:

(def ans1 (count (filter
                  (every-pred has-wovels
                              has-double
                              has-good)
                  (str/split input #"\n"))))

partition

Used in day 5, day 11 and day 13

Function allows to get a sequence of list every of which is a subsequence from original of some width and with some offset from the begining. In my case it was very useful to check any connections between sequence items.

Here is an example:

(def chmap (apply hash-map
                  (flatten (partition 2 1
                           (concat (char-range \a \z)
                                   (list \a))))))

This expression generates a hashmap with keys that are individual characters and the values that are the charecters that alphabetically follow them, except a follows z. partition takes (\a \b .. \z \a) list as input and returns sequence like ((\a \b) (\b \c) … (\y \z) (\z \a)). After that I transform this sequence to the flat list and feed it to hash-map function.

group-by

Used in day 12

Groups sequence according to return values of predicate.

iterate

Used in day 18 , day 17 and day 25

Function takes value and another function and returns a sequence that starts from the value and continues with applying function to previous sequence item to generate next one. Helps to avoid loops in all situations where you need to build next value on top of current one.

re-seq

Used in day 19, day 21, day 22, day 23 and day 25

Works like re-find but returns a lazy sequence of all consequtive matches.

Optimizing code for V8

2015-11-17T00:00:00+01:00

One of the things that I love to do the most is to take a complex system and start digging through it to discover some subtle details that reveal how the system really works. This writeup is about one of recent tasks where I wanted to understand how to accelerate execution of some arbitary code for node. I coudn’t come up with better algorythm to solve the task but suspected that the code itself could have been written in a better, more efficient way from interpreter’s point of view.

So, first I decided that I wanted to measure how fast the code was to be able to understand if I do any progress at all. console.time api is a very good way to do it:

console.time('mycode');

// ... code to be measured

console.timeEnd('mycode');

With this I can see if I do any better with the changes I introduce, the only thing is that execution time is not deterministic and it always makes sense to do multiple measurements and work with average. The other thing that can affect the measurements is the way they are done. For example, V8 engine really consists of two compilers - the former of them compiles source code to intermediate code that can be executed, and the latter works with the code that is considered to be hot (executed many times through the program run) and compiles it to the machine code. This has it’s own implications, e.g. people like to do benchmarks like this:

// ok, let's test the perf of implementation

for (var try = 0; try < 100000; try++) {
  superfunc();
}

As you see, the average execution time of this function can be affected by V8 implementation details. The other thing is that not all code can be optimized and it may really matter how you iterate on arrays and objects what language features are used and if V8 cannot optimize the code or cannot successfully guess types of variables the code will suffer from additional performance penalty.

After all this, how can one gather the knowledge of such optimizations? A very good source on this can be found here or in the awesome blog by Vyacheslav Egorov.

Additionally, profiling is always helpful and it is possible for node with help of v8-profiler.

I started my investigation from the profiler results and saw a signigicant time taken by garbage collector (50ms out of ~600) and a lot of anonymous functions. The latter were useless and I went through all anonymous functions in the code and added names to them to get meaningfull profiler report.

After some search I’ve figured out that the reason of the former was the use of array map and reduce methods. When I replaced that with ordinary loop garbage collector time reduced to 5ms on average.

Another speedup was with big object that came to be processed. On initial implementation I created another and populated with the results. Two things helped - reusing existing object and overwriting key values and using loop over Object.keys() array instead of for..in loop.

Other discovery was that function .bind method is an optimization killer and once I replaced my_reg_exp.test.bind(my_reg_exp) with function(a) { return my_reg_exp.test(a); } in a code that was executed lot’s of times I immediately got big speedup.

Yet another speedup with iteration over big array came when I replaced a call to the Object.keys() method with ‘Object.getOwnPropertyNames()`. This method also gives keys of the object but does not check the prototype and this is probably a reason of speedup.

After all these changes I got 5x-6x acceleration in the execution time without changing the algorythm or and data structures.

As a conclusion: I wouldn’t to such optimizations on the code the is executed only once throught the lifetime of the program or operates only on small amounts of data but the fact the node can have such a drastic difference in run times depending on such subtle differences really forces to keep this in mind when dealing with the code that has strict performance requirements.

FPV drone building experience

2015-11-07T00:00:00+01:00

This summer I saw some videos with FPV races that were so awesome that I decided that I want build my own drone for racing.

The least straightforward part was to actually select parts to build from. If you have no experience in hobby electronics you will be amazed by amount of what’s available on the market. There are all sorts of different frames, motors, distribution boards etc. And if you start choosing something while you’re still not sure about essential list of parts you’ll encounter a whole lot of additional options: what about led lights? If so, what kind of lights, whaterproof or not? What about onboard HD camera? And so on. All this questions take time to check compatability, voltage, connectors and other options available.

After a while I understood that the best way might be to go with bare minimum required to make drone that flies and iterate on it. There is awesome series of tutorial videos on the topic that cover all building process with just a few parts that you might want to do differently and it was vital for the success of my build.

Now, here is the list of the details that I used. I’ve split them in a sections with bare minimum on top and additional options below.

Parts list

Required parts:

H250 ZMR250 250mm Carbon Fiber
Distribution board Any board will do, this was not the best one. Actually, there are options that allow make everything without soldering at all, because they already have XT60 male plug soldered in and bullet plugs for ESCs.
XT60 Male Plug 12AWG In case you bought a distribution board that requires soldering
Tie Down Strap for 11.1 3S 2200 Battery
FlySky FS-i6 2.4G 6CH AFHDS RC Transmitter With FS-iA6 Receiver
120pcs M3 Nylon Hex Spacers Screw Nut Assortment Stand off Accessories Kit Set you need just a few of those, entire set is overkill.
AfroFlight Naze32 Acro AbuseMark FunFly Controller - Soldered version
DYS BE1806-13 Brushless Motor for Multirotor (2300KV) 24g
Afro ESC 12Amp Ultra Lite Multi-rotor Motor Speed Controller (SimonK Firmware) Version 3
HK-171 Thread Locker & Sealant Medium Strength
Polyester Velcro Peel-n-stick adhesive side V-STRONG

FPV gear. If you want to start from bare minimum this is totally not required.

Expendables:

Turnigy 2200mAh 3S 25c I bought this one, but it’s rather heavy and maybe an overkill for the start
HobbyKing® B3AC Compact Charger
Gemfan 5x3 5030 Propeller 250 Frame Probably you might want to buy lots of these

Misc stuff:

HobbyKing™ Universal Propeller Balancer, For T Style and Std Propellers
FlySky Data Cable USB Download Line For FS-i6 FS-T6 Transmitter Firmware Update The cable should enable you to use the transmitter for firmware update and for using it with simulators.

A few notes about how I chose parts.

The ideal way was to go throught every part that might be necessary and look for best options, but after a week of long evening spent by googling stuff and reading forums I decided that this strategy wil not lead me anywhere. So, checked several tutorials and selected the most popular cheap frame and combination of ESCs and motors that sort of made sense. I didn’t do any calculations about the load these motors can lift, so it was a total guess.

FPV gear was chosen because it was a) cheap and b) literally plug n play. Maybe Quanum DIY FPV goggleset will not give the best video ever but it does the job and you can really forget about FPV parts or order them later if you’re just making your first training flights.

Velcro tape is super useful to attach anything to anything. Accumulators, cameras - everything that should be detachable will benefit from it, because it is really sticky and connects parts well.

Propeller. There are different modifications of propellers but 5x3 should be the mildest option.

Accumulators. If you’re without FPV gear two accumulators should be more than enough for you and their capacity can be smaller and hence save space. And one charger is enough! If you’ll take the same transmitter like I did you can try to adopt one more accumulator to be inside of it, you can check tutorials on the net.

Assembling process

The assembling process was more or less straightforward and you can follow video tutorial I mentioned earlier. For me the main difference with tutorial was usage of power distribution board and actual setup of the controller. None of them are really hard especially if you’re comfortable with soldering.

For flight controller I used Cleanflight firmware and it’s easy enough to setup.

Few difficulties.

Radio reciever requires input voltage of 5V and I didn’t have a 2 cell accumulator. The solution was to actually plug naze32 board and use it as power source.

The other issue was with receiver was that when I started plugging wires between receiver and controller I couldn’t make it right before I stopped makeing the assumptions that channels on the receiver match channels on controller. In my cases channels like 5-6 were mixed.

Result

It was really fun and I’m eager to start studying how to fly drone better and there is also a lot of autopilot staff that I want to adress.

clj-carbonapi internals

2015-10-28T00:00:00+01:00

A while ago I decided to implement something close to the real world in clojure. The easiest way to do that was to take some existing non-toy project and try to replicate it. My choice was to reimplement parts of carbonapi and the result is called clj-carbonapi. The overall experience was so good that I decided to write a blog post with finding and pitfalls encountered during the implementation.

The first and the biggest finding is that REPL provides absolutely life changing experience and any other development process feels like a bleak shadow regarding the speed of prototyping and development especially if we talk about nontrivial code. For me the truth is that for complex tasks I sometimes don’t have any idea about how implementation should look like, what data structures should be used, what is the correct way to use new library etc. and combination of decent editor and a repl (Emacs + Cider in my case) allows to start writing some code that can be immediately tested in the repl and start building complex solution from the small pieces without a need to implement boilerplate code that should be written in any other project just to start doing something. In my case I started from implementing parser, then moved to the fetching part, added functions support after that and finally implemented graphs, server and cli interface.

As you can see, usual points that sound like first steps in most languages, just because you you need them to start seeing any results are not really necessary in clojure. When I was working on any of parts I was able to quickly iterate on the results and make things work piece by piece from small to big until I get everything right.

Having said all that I want to admit that clojure repl is not a replacement of unit tests, it’s just super cool prototyping tool. The pro of it is that you can test things continuously without any additional code to support and anyone who wrote anything with TDD techique knows that units test take a lot of time to support especially when you aim for full coverage. From the other hand repl is not a code that you can save in the repo to rerun later and tests are still needed after implementation is ready because at this moment it’s important to be sure that things will keep working as expected. In my case I wrote test for parser immediately when I decided that I covered all the possible syntax scenarios. The other benefit of the repl comparing to the unit tests is that you can actually run function from any namespace you have to get an idea how it works. Final benefit of the repl that I want to admit is the fact that you can use any functions to test results of the function call and that mean that you can play with the data with all the functions available to the core and then write real code when you’re confident with the path you want to take.

The application is split into several modules and most of them are surprisingly small. Here are the few notes on each.

carbonapi.parser

Parsing is one of the tasks that can be harder or easier depending on whether you know the right way to solve it. For parsing the right approach is to use context free grammars and clojure ecosystem provides beautiful instaparse library for this purpose. Library takes a grammar that can be written in one of two languages, compiles it and uses it to parse a string into nested list and also supports transformations, so it’s really easy to get the proper structure. I won’t go into the details here, because there are already some tutorials on the net.

The second interesting part is actual tree execution. While it’s quite possible to make a nested list that can be then evaluated there is a problem with this apptoach - all these functions won’t have any context in them and that means that AST should contain all input data (or have side effects) to make it possible to execute it with eval.

This issue is solved with postwalk function from clojure.walk package. Function takes a nested list and a function as an input and calls the function with the lists stored in the source list starting from the most nested one and goes up to the root replacing parts of the list with results of the function call. Thus it’s relatively easy to eval list manually with any logic.

carbonapi.functions

carbonapi.functions namespace provides functions to work with cabonapi metrics objects. The interesting part was that at least in simple cases I could abstract way metrics object format and allow to define operations on the data with a simple function

I didn’t implement complex cases but I think that this is not a problem of complex algorythm but just a matter of implementation.

carbonapi.remote

This namespace provides an interface to fetch data from carbonzipper or graphite instance. The json is implemented in a quite straightforward way, the most interesting part is in protobuf support. The complication with it is that protobuf has it’s own minilanguage and a compiler that turns this language in the source code in java. While definition itself was found in carbonzipper repo, the integration part erased few evening hours from my life.

The main problem was that in clojure there are two packages needed for this - clojure-protobuf and lein-protobuf and you can be as unlucky as I was to choose incompatible versions of these two packages and get a compiled java file that doesn’t play nicely with library. So, I warned you.

The other tricky thing with protobuf was that for some reasons I couldn’t see all the data in a hash returned by protobuf-load and I had to implement a function that took response parsed from protobuf input and transformed it into the structure in a format returned in json format. After inspection of compiled java file I found out that there is a method valAt that can be used to the data that is hidden otherwise.

carbonapi.chart

This part was done thanks to incanter library and JFreeChart library that powers plotting functionality of incanter.

I once again had to massage data before it could be passed to the plotting library but otherwise the library was a pleasure to work with. I easily got a graph in a new window (which is super useful in a repl usage) and at the same time I could get and image object that can be processed and sent by server to the client browser.

carbonapi.server

I used http-kit to implement server and the only note I can leave that it was easy and straightforward and integration with charts was suprisingly simple and painless.

carbonapi.core

This is a namespace that provides a top level api to show graphs and is supposed to be an obvious entry point for any repl usage.

carbonapi.main

This one is obvious entry point for cli usage. The cool part is the usage of clojure.tools.cli package that is the greatest cli parsing library I ever made.

Conslusion

Well, it was fun!

Stop pushing the specs forward

2015-09-29T00:00:00+02:00

Peter-Paul Koch wrote a post a while ago, and there was a follow-up from Nicholas Zakas. While reading the first post I found myself almost agreeing with the author. Almost, because I really think that the amount of new features is not the problem for me and I can explain why, because the reason is very simple.

Throughout my career I work on high-traffic complex sites with lots of users and all that means that all they use different browsers. From my perspective that means that I either have to use graceful degradation for new features or just you the least common denominator of all specs, and in my case it is Inernet Explorer 7 and 8. It’s not sexy, it’s slow etc., but it’s users still bring a load of money and while version 7 is slowly fading away, version 8 will be with us for another year or two. Another aspect is that we do a lot of small changes for the product and graceful degradation in pure css just does not scale. It’s relatively simple with the shadows or basic transforms, but anything a bit more complex means that there is no graceful degradation for that, the solution requires different markup for both cases. And it’s not even about flexbox (which is btw not very new, first spec was released in 2009!), it may be just calc property and a very big number if other spec that this browser doesn’t support. Even if we throw IE8 out, there is IE9 which is much better but still not perfect. There is an additional consideration: not all specs are equally good. IE6 era was perfect in sense of true research. You remember all these sites with true grids, bulletproof solutions for sticky footers and vertical alignment? One could imagine that all this should result in one true layout specification that will be flexible enough to cover all them. But it’s not the case, people make mistakes, people make APIs that are not perfect. And this means, that the thing that was once thought as solution for everything becomes obsolete artifact of the past. I remember when calc property was introduced everyone was pretty sure that that’s all we need. Er, may be also box-sizing and that’s it, all layout problems solved. And then we got flexbox, css grids, css template layouts, css columns and probably will get more. There are some specs that went out of fashion before they got implemented by major browser vendors, like css variables. I am enumerating all this just to prove my point - at least in CSS part we use technologies that are with us more or less since IE6 - floats, absolute positioning etc. And no matter how big the web platform is we use only tiny fraction of it because it is the only one that works.

Another problem with new browser features is that they are all added in append-only mode. Features can only come in, there is no way to take them out without breaking the web somewhere. And that means that even unpopular or badly designed specs are here forever to annoy users and bloat browsers. Time passes and now it’s more and more clear that even cascade part of CSS was not the best possible solution. And even more, if you ask me, what relatively new CSS specs are really worth existing, I would mention media queries and transitions and this is it. But I can easily mention at least two libraries types that really changed the landscape of styling - css grid libraries and css preprocessors. There was no need to write a spec about them and implement them for evey vendor, because these two technologies happily live on top of existing standards.

If we consider JS part of web platform it appears that the situation is a bit better in some parts but fundamentally the same. Features keep being added, but we still have to send ecma 232/3rd ed. to browser because that is the only version everyone supports. And again we are years behind from latest and greatest and all innovation again happens in libraries - we have all sorts of languages compiled to javascript, we have transpilers or even macros that give us the latest version of ES standard in any browser without a need to implement all that in the browsers. Even module loaders happily existed as external libraries for years before we got ES6 modules spec, and this spec still has whole bunch of questions to solve - aliases, bundles, slices and a lot of features that existing module loaders already have.

One would say that current fragmentation only exists because of IE, and once these guys figure out how to quickly update their browser the problem will be solved. Certainly it will help, but mobile fragmentation still exists and some browsers even start to implement -webkit- properties, because of total dominance of different webkit based browsers in some markets.

So, my point is that new standards have a very long way to go before any wide adoption for natural reasons and the only way to get through this mess is to abstract away and build something on top. And this is the exact reason why we see a real explosion of influentual libraries like Bootstrap or React.JS. They just work for majority of cases, they fix big problems now without the need to wait for indefinite future. But do I want to see them or their approach to be adopted in browsers as a spec? No, because that will kill the major benifit of the libraries - you can use them as long as they fit the purpose and you can throw them out if something better appears on the market.

So, what is the conclusion of all this writeup? I think that the future is not after new platform extensions, but in libraries that will provide standard ways to solve standard problems. Dialogs, lightboxes, calendars, layouts, endless battles about module formats, server communication - that’s what bothers most of people and wastes billions of work hours and solving these issues will push the web forward for real. And there are plenty of more or less successful attempts to solve these issues.

Img.js - first optimizations

2015-03-30T00:00:00+02:00

This winter I studied the fascinating course about image and video processing by Guillermo Shapiro and decided to make my own javascript library with bells and whistles and also try a few approaches to code that were not battle tested by me.

From the start I wanted to get several things:

es6 library code with umd es5 code at the end so that it could be used in any browser environment.
monadic approach to api, so that all the operations on images can be both synchronous and asynchronous, and we still get all this beautiful chaining api.
internal api, that will consists from the set of operators, that can be composed together before applying to the actual image. The rationale is that if you for example make two consequent inverse operations than the library should not process image at all in this case. And also it should be very fun if I can get that far.
very little public api that is really public only for transformation plugins. In ideal situation I want to be able to build the library for any number of transformations if I want to and don’t ship everything under the sun just because it’s easier. And who nows, may be I will make a img.jspm some day :)

First two goals were relatively easy ones and probably I’ll write about them in next post, but the last one required at least some understanding about what kind of transformations I will want to make with the image so that I could start defining some solid api.

The first and the most trivial transformation is identity where you transform the image in such a way that the result will be identical to the source. I got there very easily and decided to move to the next one after that and use the identity filter only in cases if I’m changing core api and want to test on something that doesn’t contains potential algoritmic errors for sure.

The next easy filter was grayscale. The first thing to notice there is that there is no such thing as canonical grayscale. If you check this post than you will see that there are at least three of them and there should be more because the only requirement is equality of R, G and B values for each pixel at the end. I took the path of luminosity approach. I have no idea about where the coefficients come from, but here is the first naive implementation:

lift('grayscale', function grayscale(data) {
  var new_data = create_image_data(data.width, data.height);
  var cur_pos, new_pos;
  var luminosity;

  var arr = data.data;
  var new_arr = new_data.data;

  for (var x = 0; x < data.width; ++x) {
    for (var y = 0; y < data.height; ++y) {
      cur_pos = (y * data.width + x) * 4;
      new_pos = (y * data.width + x) * 4;
      luminosity =  0.21 * arr[cur_pos] +
                    0.72 * arr[cur_pos + 1] +
                    0.07 * arr[cur_pos + 2];

      new_arr[new_pos] = luminosity;
      new_arr[new_pos + 1] = luminosity;
      new_arr[new_pos + 2] = luminosity;
      new_arr[new_pos + 3] = arr[cur_pos + 3];
    }
  }

  return new_data;
});

Here is the client code to run this:

img('./test.jpg')
  .grayscale()
  .outputTo(document.querySelector('.output'));

The facts to mention here is that:

the code is asynchronous
I don’t care about browser compatability at all right now and it was one of the starting design decisions for the library.

I’ve been looking on the grayscale code again and again and it was apparent that it’s nothing more than just matrix multiplication.

Wouldn’t it be nice if I’ll just right this:

lift('grayscale2', function grayscale2(data) {
  var new_data = clone_image_data(data);

  return multiply_matrix(new_data, [ [0.21, 0.72, 0.07, 0],
                                     [0.21, 0.72, 0.07, 0],
                                     [0.21, 0.72, 0.07, 0],
                                     [0   , 0,    0,    1] ]);
});

Here is the first implementation of multiply_matrix function:

export function multiply_matrix(data, matrix) {
  if (matrix.length !== 4 || matrix.filter( (el) => el.length !== 4 ).length > 0)
    throw new Error("filter matrix can be only 4 by 4");

  let max = data.width * data.height;
  var tmp = [];

  for (let pos = 0; pos < max; ++pos) {
    let real_pos = pos * 4;

    for (let comp = 0; comp < 4; ++comp) {
      tmp.push(
        data.data[real_pos] * matrix[comp][0] +
        data.data[real_pos + 1] * matrix[comp][1] +
        data.data[real_pos + 2] * matrix[comp][2] +
        data.data[real_pos + 3] * matrix[comp][3]
      );
    }

    for (let comp = 0; comp < 4; ++comp) {
      data.data[real_pos + comp] = tmp[comp];
    }

    tmp.length = 0;
  }

  return data;
}

The main problem with it was the performance (grayscale2: around 350ms and grayscale around 125ms on my MBP for 1024x768 image) and I was not really sure was it about algorythm or about js interpreter itself.

Searching the internet revealed that matrix multiplication is not a really easy task to optimize in terms of performance and I decided to focus on js internals.

Profiler showed that Chrome was really busy with garbage collecting and setting length of array to zero costed me around 100ms. Wow.

Ok, I’ve tried to assign new array at the end of each loop pass:

var tmp = [];

for (let pos = 0; pos < max; ++pos) {
  let real_pos = pos * 4;

  for (let comp = 0; comp < 4; ++comp) {
    tmp.push(
      data.data[real_pos] * matrix[comp][0] +
      data.data[real_pos + 1] * matrix[comp][1] +
      data.data[real_pos + 2] * matrix[comp][2] +
      data.data[real_pos + 3] * matrix[comp][3]
    );
  }

  for (let comp = 0; comp < 4; ++comp) {
    data.data[real_pos + comp] = tmp[comp];
  }

  tmp = [];
}

The perfomance fell to miserable 550ms but very fun fact was that profiler didn’t show anything specific and garbage collection was reducde by and order of magnitude.

Ok, let’s try to go from the other side and not reset the length of array at all. Memory is cheap.

var tmp = [];

for (let pos = 0; pos < max; ++pos) {
  let real_pos = pos * 4;

  for (let comp = 0; comp < 4; ++comp) {
    tmp.push(
      data.data[real_pos] * matrix[comp][0] +
      data.data[real_pos + 1] * matrix[comp][1] +
      data.data[real_pos + 2] * matrix[comp][2] +
      data.data[real_pos + 3] * matrix[comp][3]
    );
  }

  for (let comp = 0; comp < 4; ++comp) {
    data.data[real_pos + comp] = tmp[real_pos + comp];
  }
}

And it was the first success: around 260ms! Much better but still not as good as the original thing.

I decided not to give up and started looking for the hints about v8 interpreter internals that can give me the speedup. The most interesting video I’ve found gave me all sorts of hints about how I can shoot myself in a foot in terms of performance but I didn’t get much after discovering the fact the multiply_matrix function is getting deoptimized constantly. Maybe I’ll return to this later, but one cool fact that I’ve discovered is that you can get the fresh instance of the Chrome browser by just starting it with different working directory and it will hold the version that was with it right after install before updates happened. Also you can pass parameters right to the v8 engine and get all sorts of debug output.

/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --user-data-dir="/tmp/chrome_dev" --js-flags="--trace-opt --trace-deopt --trace-bailouts"

Here –user-data-dir option will give the fresh instance (even if there is a running one) and –js-flags will be passed to v8.

The main point for the talk is that the more information you give to browser about types in your code the more speedup you may theoretically gain. I remembered that typed arrays were introduced for all sorts of low-level number crunching and found a very good article on Mozilla Hacks.

I decided to start with just Uint8ClampedArray:

var buf = new ArrayBuffer(data.data.length);
var buf8 = new Uint8ClampedArray(buf);

for (let pos = 0; pos < max; ++pos) {
  let real_pos = pos * 4;

  for (let comp = 0; comp < 4; ++comp) {
    buf8[real_pos + comp] = (
      data.data[real_pos] * matrix[comp][0] +
      data.data[real_pos + 1] * matrix[comp][1] +
      data.data[real_pos + 2] * matrix[comp][2] +
      data.data[real_pos + 3] * matrix[comp][3]
    );
  }
}

data.data.set(buf8);

It gave me blazing 120ms, as fast as original, but I was curious about using Uint32Array and decided to try it:

var buf = new ArrayBuffer(data.data.length);
var buf8 = new Uint8ClampedArray(buf);
var buf32 = new Uint32Array(buf);

for (let pos = 0; pos < max; ++pos) {
  let real_pos = pos * 4;

  buf32[pos] = (
    ((data.data[real_pos] * matrix[3][0] + data.data[real_pos + 1] * matrix[3][1] + data.data[real_pos + 2] * matrix[3][2] + data.data[real_pos + 3] * matrix[3][3]) << 24) |
    ((data.data[real_pos] * matrix[2][0] + data.data[real_pos + 1] * matrix[2][1] + data.data[real_pos + 2] * matrix[2][2] + data.data[real_pos + 3] * matrix[2][3]) << 16) |
    ((data.data[real_pos] * matrix[1][0] + data.data[real_pos + 1] * matrix[1][1] + data.data[real_pos + 2] * matrix[1][2] + data.data[real_pos + 3] * matrix[1][3]) << 8) |
    ((data.data[real_pos] * matrix[0][0] + data.data[real_pos + 1] * matrix[0][1] + data.data[real_pos + 2] * matrix[0][2] + data.data[real_pos + 3] * matrix[0][3]))
  );
}

data.data.set(buf8);

I’m not detecting endiannes here just measuring speed. And measurements gave me the incredible 20-25ms, five times faster than original function!

Probably there is a way to make even this magic number lower, but I decided to stop there for a while.

Another speedup I got at the image rendering at the end. At first I wanted to get a proper img tag at the end and my code looked like this:

lift('outputTo', function outputTo(data, el) {
  var c = document.createElement('canvas');
  c.width = data.width; c.height = data.height;

  var ctx = c.getContext('2d');
  ctx.putImageData(data, 0, 0);

  var node = new Image();
  node.src = c.toDataURL("image/png");

  el.innerHTML = '';
  el.appendChild(node);
});

Chrome on averaged spend up to 200ms on this and at least half of this time was spend on the toDataUrl method execution. Firefox behaved even worse with 600ms. It was too bad and I decided to drop image part for now and output just canvas:

lift('outputTo', function outputTo(data, el) {
  var c = document.createElement('canvas');
  c.width = data.width; c.height = data.height;

  var ctx = c.getContext('2d');
  ctx.putImageData(data, 0, 0);

  el.innerHTML = '';
  el.appendChild(c);
});

And the timing went to 5ms at firefox and less the one ms in Chrome.

With all these optimisation the image processing part became barely noticable which is very good and opens space for more advanced filters that will run in a reasonable time now.

Adopting Microservices at Netflix: Lessons for Team and Process Design

2015-03-11T00:00:00+01:00

Article tells the story of Netflix’s approach to the development process and deployment. There are some really good parts like making assumtions explicit and validating them from time to time. From my own experience I’ve seen that kind of things multiple time. Organizations get bigger, people get older care more about vacations and overall stability and care less about company future or new technologies at all.

I cannot really judge the part about using cloud solutions everywhere, but continuous delivery process and independent serivces, that are somewhat easy to deploy is what I experience now and I don’t want to go back.

Learning how to learn: my walkthrough

2015-01-26T00:00:00+01:00

I decided to write down my own thoughts and conclusions from the course “Learning how to learn” as a final project. My own part will represent some ideas that I have and that provide more context to the course on my opinion.

First of all, the motivation part. Why do we learn? What do we learn? This term has a very broad definition and it’s very easy to end up learning lot’s of things of very little importance or use for us. The world around us is fascinating and there are tons of things that are worth knowing. In reality the number of these areas is even bigger and every learning subject hides lots of books, videos and courses behind it. You can take photography, drawing, biology, artificial intelligence - any field you can imagine, and it will be really huge and can be split into smaller independent pieces that are huge on itself. When we learn we’re trying to make ourselves better in this sense or another and it’s very easy to start learning everything at once and it’s obvious that the possibility of learning everything at once is much lower than in situation when you have only one subject that you can master. So, failure is very much possible and this might bring the illusion that the ability to learn has vanished and basically nothing new finds its way into the head, which is just not true.

As was stated in the course we all have two types of memory - shortterm and longterm one, and what happens is that we place some initial knowledge into the shortterm memory and then switch to another subject which will mostly replace the previous one with new information. The solution is to have a focus and it’s very important on my opinion. The focus in this situation is an ability to prioritize items that you want to learn and prevent yourself to learn anything that is not a top priority.

How to get the focus? The most important thing on my opinion is to have an idea what exactly do you want to reach at the end or at least to know the direction where you want to go. To have the goal or to feel the meaning is important in all areas of life, but it’s really important in learning, because it will help to structure studies and to define a roadmap for future.

These are the most important things that I want to add on my own. If we talk about individual courses or books that are to be mastered I would like to introduce my own favorite set of ideas from the course, that I treat as very important.

The two most important concepts that I got from the book are chunking and the notion of long and shortterm memory. These terms are not actionable by themselves but different methods described in the course help use one concept or another.

Chunking is is a process of forming self-contained bundles out of any ideas and terms so that they can be used on their own in different context without figuring out all underlying details every time.
Shortterm memory is the part of our memory where we hold the information that we found out right now and we can easily forget.
Longterm memory is the general storage of our knowledge. All things that we know very well end up in this type of memory.

So the process of learning from my understanding is a continuous process of forming chunks and putting them in the longterm memory where we can use it.

Which of the methods shown in the course are most helpful on my opinion and from my experience? I would say that they are practice, (self)explanation and the ability to switch off.

First of all, practice. The fact we understand something doesn’t necesary mean that we know it or can do it. Most of the times our understanding is superficial or doesn’t take into account all aspects of reality, but that matters a lot and not only for learning new subject but also for using it in real life. I would take the example that is closest to me at the moment. When we talk about algorythms in robotics there are lots of layers in them. You can learn high level planning solutions, but if you try to implement them, you will find that they do not just work by themselves, there are several layers below them, each closer to the hardware than the previous one. On the top level you can produce command clean(floor), but level below there is a whole new set of problems that include navigation on the middle level and actual interaction with the hardware on the level below that, and the complexity of the subjects is not evident when looking on the topmost algorythms. Given that the practice does not only allow to get the ideas from the shortterm memory to a longterm one, it also provides lots of valuable insights of how this idea interacts with others and can probably suggest the next steps in the studies and maybe even can influence on future projects that you may have.

With all these properies practice is helpful in conquering both chunking and putting subject in a longterm memory for later wild use. Practice really helps to get the feeling of the constraints, limitations and dependencies of the ideas. The important point is that not every type of practice is helpful. It’s very easy to solve simple problems but only finding solutions to difficult ones makes us smarter.

The second important part for me is (self)explanation. I would say that it’s a somewhat natural part of practicing, but if with doing exercises you can find the limitations of the methods and ideas, with explaining ideas you can immediately see the parts that are still weak and are needed to be studied better.

The last thing is an ability to switch from the subject and be able to keep rhythm and relax. If we find the goal that we strongly want to achieve it’s very tempting to try to bruteforce it and learn it through several very long sprints by devoting all energy to learning. The sad truth for me is that whenever I tried to do that I failed because I got exousted very soon after beginning. The correct way that really works for me is to develop the rhythm of doing something and try to do my best during scheduled hours and then switch to some very different task such as movies or (especially) physical exercises. A believe that doing sports involves very different parts of the brain that studies and sometimes I can really turn myself into kind of thoughtless mode when I just do what I do and all thoughts are shifted to the background. I think it exactly correlates with the diffuse thinking mentioned in the course. But for me it’s not only the way to process information in the background. Switching activity is also a way to relax and besides allowing as to grow in several directions it also allows to fill the progress in different things you’re doing and that helps to keep the motivation on the higher level.

At the end I want to dwell a bit on the procrastination. I think all people face it at some level or another. Sometimes the task is not that pleasant to do or it’s just too big to grasp or we’re just tired or didn’t get enough sleep. As was said in the course, for everyone there are cues that trigger something within us sending us to spend another twenty minutes in the facebook. But what’s good is that we can build them to help us to work. Pomodorro technique is especially helpful in this sense that it puts the focus on the work. It may seem that 25 minutes is not that big time, but if you try to setup timer during the work, you can suddenly find out that in some days it can happen that you accomplish just two or three sets of pomodorro. Even with this technique there is still a room for emprovement and it’s about planning. Simple todo list will help a lot, because you there will be no excuse that there is nothing to do and even it you are not in the best shape in this particular day you can just take one task after another and complete them even mechanically - anyway you still make a progress and you wouldn’t otherwise.

I think I’ll finish here. All the things described here are a prototype of my ongoing learning experiment. I really struggle to implement it in full scale in my life and will be very glad if anyone can find this document helpful in any sense. Thanks for your attention.

Taming the Asynchronous Beast with CSP in JavaScript

2014-09-09T00:00:00+02:00

Another amazing article by James Longster, that continues his series of blog post about web development from future, where he speaks about all kinds of features, techniques and new abilities of Javascript language that completely change the way how we can deal with common problems during the development.

One of the problems is constant dealing with events, wich leads to rather complicated code that balances between different events by containing intermediate state somewhere and the more complex interaction gets the more complex code we get. Another source of asynchronous complexity is actual async part which comes from any ajaxy activity.

Js-csp is a neat way of dealing with all these two issues in a uniform way. The way it unified if that all the differences are abstracted away by channels where any value can be put in and read from. The key difference, that these operations are blocking with proper try/catch support etc. Hence you get several blocking blocks of code that actually only block themselves and communicate via channel messages.

Number Words and Number Symbols: A Cultural History of Numbers

2014-08-17T00:00:00+02:00

I had the russian edition. The book consists of several sections and begins with author’s explanation about how numbers had possibly emerged in the languages of different nations. They appeared not at once, but one after another anf often not sequencially but when they were needed. That’s why round numbers (tens, hundreds) emerged earlier then their lower non round neighbours and the names of round numbers affected the names of their neighbours too. In some cases people even didn’t need the actual names for the numbers and used signs or special names connected with the field they were used at. Right now we all count in the decimal system, but much earlier people have used names based on five or on twenty. Even the definition of the number was evolving. Not only zero wasn’t treated as a number, one was also not treated by europeans as a number for a long time.

There are myriads of different facts represented in the book and all they are accompanied with lots of photos and cross references. The density of them sometimes is so high, that it was a bit hard to get through all of them while reading. Also, some examples or descriptions of the calculation techniques and algorythms are not well defined and may contain errors (may be it’s only true for russian edition) that it takes significant time to get it right. For example to really understand how calculations were done by Sumer I just took anothe book (“History of Mathematics” by A. Yushkevich).

The big advantage of this book is that during exmplanations author constantly shows connections with current languages and shows how different words or terms were born like bankrupt, bureau, the names and symbols of currencies etc.

It’s very interesting to read section devoted to calculations and numbers usage in Middle Ages. We are so used to arabic numbers (that are actually form India) that it’s hard to believe that in the middle ages most of european countries used roman numbers for writing and special calculation table for actual calculations, because roman letters do not provide any machanics to use them in the calculations directly. These tables and their analogs were wide spread everywhere from Japan to Western Europe and were used even in the twentieth century.

It may also be a revelation that some nations used non-decimal numbering systems for the calculations, based on twenty or sixty, and it affected the way the calculations were done by them. The consequence for us now is that we have 60 seconds in a minute now.

The last cool thing that I want to mention is how calculations were made when the paper was expensive and not widespread. All sort of counting rods were used in the Europe and one type of them was called Split tally sticks and was used by British exchecquer until 1826. It was used for tax collection or debt recording. Every stick with the number of tax to be paid or debt to be returned was split into to part and by joining them again one could verify that the payment is correct and should be done.

The book has a lot of useful information and it is a drastic difference comparing with most of modern popular science books that usually have the same volume but contain just few facts and just repeat them through the pages again and again in different variations.

I highly recommend this book for all those who are interested to know more about how we ended up with all the numbers and the things around them that we have now. The only advice I can give is not to dig very deep into the sections connected with the representation of numbers in different languages. This section has so many comparisons and crossreferences between different language that it really doesn’t give a lot to the general idea but it really takes a lot of time to understand the whole thing.

As a conclusion I want to say that I was surprised to find very little information about the author in the internet, and there is actually rather small number of references to this book, at the same time it’s quite old. All this raises some suspictions, that this book is not a classic one on the subject, so probably it may differ from “official” points of view and it may be worth to double check any information of any importance in different sources before actually using it anywhere seriously.

Getting it done with Haskell

2014-08-09T00:00:00+02:00

A presentation about real-life experience with Haskell development. The language is more than 20 years old but still the real applications of the language are rather rare, so such links may be very useful.

I don’t know a lot about language itself, so some of the slides are rather cryptic to me, but interesting parts are athe beginning where the overview of the literature is and the end of the presentations where one can find general suggestions about how to sturcture the haskell development if you want to do it.

May be worth looking again in some future where I and Haskell will meet together.

IBM Chip Processes Data Similar to the Way Your Brain Does

2014-08-08T00:00:00+02:00

IBM has just presented the computer chip with the fundamentally new architecture that is really different from Von Neuman arctitecture in all the ways. It’s still built with transistors but at the end it’s a hardware made neural network, where these transistors work as perceptrons or synapses.

On the tech demo it was demonstrated that such a chip can do image recognition from a live video stream with a very little energy consumed and at the same time the decent notebook could only deal with 10 times slow video stream and consumed 10,000 more power.

Since it’s a neural network it requires very specific programming. It has not distinct processor and memory, because such network does both. And to program it correctly it’s required to program how parts blocks of the chip should be interconnected and how much influence they will have on each other. And then one should run the code in the chip simulator and only after that the code could be run on chip.

So the chip is really great in terms of future potential but provides a very big challenge in terms of programming because it’s so different from anything else.

Building the Firefox browser for Firefox OS

2014-08-08T00:00:00+02:00

Long and interesting story of a project that ended up as a firefox OS. Probably it’s only a part of the full picture, because author only talks about building browser for the system, but it’s still very interesting who it all went from a few men project with a quick and dirty prototype to a fully featured browser that will later integrate with the system almost entirely to diminish the borders between apps and websites.

The story is interesting also in sense that the development happens really around the world from Paris to San Francisco and Tai Pei which reminds my somehow the novels of Neal Stevenson.

The other thing worth looking at is a number of improvements that was made in Gecko in order to make it possible to beal fully featured browser web app.

The wind rises

2014-08-06T00:00:00+02:00

Animation film by Hayao Miyadzaki. Film is absolutely perfect if you think about it without historical context of that time and I would highly recommend to watch it carefully instead of reading another one big article about how to fight ro live with procrastination.

The main figure, Jiro Horikoshi dreams about planes all his childhood and spends all his animated life by creating them and devotes all his time to his dream. His desire to build the perfect plane is so high that when he gets the mission to build a war plane he thinks more of how to build it than on the fact that Japan is going to attack surrounding contries to expand it’s influence. Or probably he approves these actions, but I think that Miyadzaki purposely omitted this in the movie.

If we will put the war away from the story then at the end we get a story of a man who loves his job to death who dreams about it even when he doesn’t work. And who loves his wife but even given that she’s very ill he can get from his obsession with planes and spend a bit more time with her.

And it’s the situation that make the film so magnicient to me. Jiro works on his plane even when she comes to him to spend her last days with him. The illness is on the final stage and what happens is that he just accepts this fact and continues to work on his plane. His wife Nahokoalso accepts this but also accepts the fact the even these days Jiro spends most of his time pursuing his dream. We even see that they both see the pain of all this, but he doesn’t stop working even through the tears.

The movie reminds me the life of such people as Sergey Korolev who work to make their dream come true and even big struggles cannot stop them from doing it.

Refactoring in Java, Scala, and Clojure

2014-08-06T00:00:00+02:00

Talk that is interesting only because it’s an example of attempt to somehow compare expressiveness of three JVM languages - Java, Scala and Clojure. Author represents a simple problem with solutions on all languages and then modifies the problem to show how different languages can easily adopt to them.

Author shows that Java solution is always bigger in terms of LoC and requires the developer to work more on the type definitions than on solution of the problem. The thing is that problem described in the talk is a toy one, so it has nothing to do with real big systems and gives only feel of how it can be in different languages. The other fact that decreases the value of the talk is that at the end author says that although java looks very verbose and awkward comparing to two other languages he still use it the most and will continue to do it for any real projects in future. Given that there is a doubt that he really believes in what he is talking about.

Out of the Tar Pit

2014-07-28T00:00:00+02:00

Article is rather abstract at start and begins with defining what the accidental and essential complexities are in terms of program systems. The aproach is rather radical, because the essential complexity is treated only as a complexity that directly derives from the user specifications, so all language and infrastructure complexity is accidental which is some sort of bitter true. The proposed approach is to eliminate all possible accidental complexity if possible and separate remaining from so called essential logic and state si that they should not ever know about it’s existance. The proposed way to define such a system is to use relational model which can help defining all the system essential requirements and constraints in a purely declarative manner and just provides endpoints to extend the system with accidental complexity parts if needed for performance or usability reasons.

Station Q

2014-07-25T00:00:00+02:00

This is article is related to one of the Microsoft Research project but doesn’t look like ordinary promo page from Microsoft. The purpose of article to clarify the purpose of attempts the Microsoft is doing in the field of quantum computing. What they are trying to do is to develop a scalable method to build quantum computers. Today a lot of labs have an ability to make some kind of quantum computation, but noone can build a stable computer and the reason is that any kind of distraction to one of qubits during computation destroys all the information, and they try to apply a topology to the field to make a protected version of qubit that will be able to resist distraction or at list its distraction will not lead to the failure of the computation. In 2000 a theoretical particle Majorana was proposed to exist and later one of the researches got evidence of their existance. If it’s true it should be more easy to build a protected qubit. It’s just a chance but if it succeeds we will get a scalable way to build quantum computers. Why is it imortant? Theoretically quantum computers are exponentioally more performant than ordinary ones, so the problems that can take thousends of year could take days or even hours and we will see a real explosion of scientific discoveries

Meet Ray, the self-driving forklift that is parking cars at a German airport

2014-07-25T00:00:00+02:00

Dusseldorf airport adopted the first system for automatic car parking called Ray. Person can just pay for the parking and the robot will lift the car and move it to the parking space. Ray device is fully autonomous and use lasers to navigate. Author claims that with this system the parking can be used up to 40% more efficiently with it than without

The Secret Government Rulebook For Labeling You a Terrorist

2014-07-23T00:00:00+02:00

Very detailed article about how US government places people (in us and world-wide) in the lists of the terrorists suspect. Most worries are connected with the fact that the conditions to get in a such list are very wide (the term reasonable suspicion is declared, but it has a very wide definition, a lot of exceptions and if that also does not help, there are exception, that allow to place e person in a list even without any real facts), there is no way for a person to figure out if he is in such a list and there is no public procedure to get out of this list. Moreover existing mechanisms have nothing to do with the official court system and exists inside of military or intelligence agencies without any access from outside. If person was placed in such a list it immediately gets problems with transportation, job and any other area that has something related to security. If we think about other articles that show the scale of monitoring systems developed by US government it becomes really scary and cyberpunk novels become really close to the reality. The last fact that is worth mentioning is that if the person gets in the list all government or government related structures a forced to collect all the possible data of such a person and interestingly it’s not about military and intelligence agencies but also the organizations that should not have any connection with politics and monitoring people like USAID.

Riding the Juggernaut That Left Print Behind

2014-07-21T00:00:00+02:00

Another article that pokes around the question that print media becomes more the part of the past. It cannot compete with social networks in terms of reaction on events, because new tweets and videos go online almost immediately. The interesting thing that is mentioned by author is that Internet provides different style of reading than print. With internet you just can get enough and always switch to the new content. As a consequence, one can hardly remember the information that he got from all the links he has read today. Probably, it’s not needed, but what is the reason for reading it anyway?

Love People, Not Pleasure

2014-07-21T00:00:00+02:00

Third NYTimes article in a day. This one deals with the question of one’s happiness, no more no less. One statement is that it’s not only happiness and unhappiness, it’s happiness and all shades of unhapiness. Fame, money, sex even make the person more unhappy than he was before, and it’s not the right way to go if the only thing you want to achieve is a happy life. Author states, that all these things work more as adrenaline than a cause for happiness, the more you get it the more you needed the less happy you are. And the only way to avoid this trap is to stop trying to reach happiness though this things.

Letter of Richard Feynman to the graduate

2014-07-21T00:00:00+02:00

The letter Richard Feynman wrote to one of his graduates to explain that it’s worth to work no only with world-scale problems but with all problems that you are capable solving of. If you’re not noticed by most of the world, you’re still known and loved and recognized amongst friends, relatives and colleages and you can always do something about that. Probably it’s worth to print this letter and to hang it on the wall in my room. The stie is worth additional exploration

Africans Open Fuller Wallets to the Future

2014-07-21T00:00:00+02:00

Article raises the discussion around current growth of african economies. The country are still mostly poor, but their economies are booming and western companies invest big money not just in mining but also in foods and manufacturing. Also the number of people with middle income raises and they are not the only one close to the ruling party. South Africa still performs the best, but Nigeria also shows rapid growth. They even launched several satellites which is almost unique amongst african countries. In the next century the whole region can become a driver of growth of the world.