23 August 2021

Clojure CLI Tools in Debian - GSoC 2021 Final Report

NOTE: this blog post is based on my "Clojure CLI Tools in Debian" GSoC 2021 project Final Report.

Intro

My name is Leandro Doctors (“allentiak” on IRC), and I’ve been the GSoC intern working with the Debian Clojure Team during 2021. This is my final report. You can also check my original proposal and my first report.

Summary

Whereas the raw data may not sound by itself very positive, my personal conclusion is. This is, whereas I didn’t fully finish the required deliverables envisioned in my original proposal, I do feel I am much closer to, eventually, becoming a Debian Developer. So, by all means, I consider this project has had a positive outcome.

Project

The goal of the “Clojure Build Tools in Debian” project was to provide Clojure Debian users with some of the latest advanced build tools and libraries the upstream Clojure developers have been lately working on. These include tools.deps.alpha, a library for dependency graph resolution and classpath building, and the CLI tool clj, for REPL interaction. If time permitted, I was also to improve the quality of both new and existing Clojure packages, and the overall Debian Clojure packaging process. My mentor was Louis-Louis-Philippe Véronneau, and my co-mentor was Utkarsh Gupta.

Motivation

Why this project? On the one side, if you’re a Clojure lover like me, you may have noticed that the Clojure experience in Debian is, as of mid 2021, well... still quiet limited. Additionally, this project aligned with my own background in Free Software community building and my research interest in Peer Production.

When I mention how limited today’s Clojure experience in Debian is, I can see two reasons for this, deeply intertwined. The first one is that there currently aren’t many Clojure-specific packaging tools in Debian (such as a clojure-debian-helper). The second reason for which we only currently have a suboptimal Clojure experience in Debian, and probably the root of the previous one, is that many core build tools and libraries for the language have not simply been packaged yet. My project aimed to attack that seemingly root cause.

As I said, another reason for me choosing this project is my own experience as the Co-founder and Leader of, probably, the first Free Software Community experience in my hometown of San Juan, Argentina. That interest in Free Software evolved in a first PhD attempt in what is now known as the field of Peer Production. A subject that has lived within me as a research interest during my day job at a University.

Being a Clojure fan, it felt only logical combining all those interests somehow. And this project seemed like the ideal combination.

The Debian Clojure Team

I’ve been working with a small, yet very warm team. The current incarnation of the Debian Clojure Team exists thanks to the hard work of three people.

Elana Hashman (aka “the Clojure necromancer”), revived the team around three years ago. Later on, the team gained the invaluable presence of Louis-Philippe Véronneau and Utkarsh Gupta (my mentor and co-mentor, respectively).

Together, these “Three Musketeers” have maintained the team alive, allowing us, Debian users, to enjoy Clojure.

Status

During the first part of my project, I mainly worked on learning the basics of Debian packaging, and got my first package uploaded. I have to thank Louis-Philippe, Utkarsh, and Elana for their immense patience and support during that part, as it took me quite some effort grasping the basics of Debian packaging.

During the second part of my project, I worked on my last packages, and almost completed the originally required scope of the project. I only have to finish working on the transition from the currently provided set of packages (based on a Debian-specific clojure runner) to the newly provided upstream clojure and clj runners.

Unfortunately, I didn’t have much time left to start working on the opportunities for improvement already identified by the Debian Clojure Team originally outlined in my proposal. Whereas I did update one older Clojure package not built using leiningen (tools-data-xml-clojure), I didn’t write any Lintian tags to make Clojure packaging in Debian more robust, nor worked towards the automation of Clojure unit tests in autopkgtests via autodep8.

Deliverables: Data vs. Conclusions

If we are to talk about deliverables, we should start with the data. According to my original proposal, I was required to provide both new and updated Clojure packages accepted into Debian “unstable”, and updated Clojure packaging documentation. Additionally, if time permitted, I was to also provide new Clojure Lintian tags merged by the Lintian maintainers, and new Clojure autodep8 scripts merged by the autodep8 maintainers. Whereas I partially accomplished both required tasks, I didn’t manage to start working on any of the optional deliverables.

When looked in isolation, those numbers may look somewhat disappointing for some people. However, I can draw a much more positive conclusion. Why?

Firstly, GSoC is supposed to be a learning experience. Moreover, as I said in my original proposal, “I approach[ed] this project as a great opportunity to, finally, start my journey towards becoming a Debian Developer”. In that sense, I consider the time invested into this project fruitful. In this way, I have learned the basics of packaging, how to interact with the Debian Clojure Team, and and already got my first packages accepted. Plus, I’m looking forward to continuing to work with the Debian Clojure Team so I can attain the original scope of the project. Therefore, all things considered, I can consider this experience as a moderate success.

Lessons Learned

Technically speaking, if I have learned one thing during these weeks, is that packaging, although easy to be underestimated, is by no means a trivial process. As any Debian Developer surely knows, the onboarding process can take some time. Plus, what is easy for some people, can be difficult for others. In my case, this was quite evident. Whereas I can speak several languages and learning new ones takes me little effort, grasping the basics of packaging took me (literally!) blood, sweat and tears. Indeed, the packaging learning curve was quite steep for me.

That being said, I did learn a thing or two about packaging. So, if I managed to get here, I’m sure many others can. It may take them more or less time than what it took me, but learning (at least the basics) of packaging is an achievable goal.

Technical skill learning aside, I value very highly the non-technical skills I have so far improved during this project.

For instance, I also learned that it can take some time to adapt to real-time online communication. Before this project, remote working meant either exchanging emails or getting into video or audio calls, with a low emphasis on chat-based interaction. Early on, I realized that the Debian Clojure Team interacts almost exclusively via, well... chat! And those two approaches are very different indeed. It has taken me some time to adjust, but I’ve improved greatly in this aspect as well.

Finally, improving my time management skills has been also a key part of this process. Whereas I had already been working remotely for over a year and a half already, my day job is not so interaction-dependent as this project (specially in the beginning). So it took me some time to adapt to this way of working, and to plan my workload so I could use those waiting moments to advance in other parts of the project. Still a lot to improve here, but improving nevertheless.

Acknowledgments

I first have to thank upstream. More specifically, one of the upstream developers of the clojure-tools, Alex Miller. Everytime I needed specific information on what do specific parts of the Clojure CLI tools’s codebase do, tools.deps.alpha do, he popped up a reply in a matter of hours. He has shown genuine interest in the success of is project during by carefully replying to my emails with detailed explanations of code intent and form, both in private and in public conversations. Thank you for all that, Alex!

Let’s move on to the Debian Clojure Team.

First, Elana. I thank Elana for her initial openness when I first contacted her about this idea. It was *her* who initially contacted Louis-Philippe so he would become my mentor. I wouldn’t have started to work on this project if it wasn’t for her. Plus, she provided quite a piece of advise in more that one ocassion. So, thank you very much for all that, Elana!

I also thank Utkarsh, my co-mentor, for his overall technical advise. And a special mention to his initial help to setup my Matrix client for OFTC chat. At that moment, it was *him* who took the time to help me in real time so I could solve that problem. So, thank you very much for all that, Utkarsh!

I finally have to thank Louis-Philippe, my mentor, for his patient guidance during the whole process. His dedication and hard work has been *instrumental* for my progress. And a special mention for his tolerance with respect to some unforeseen personal circumstances I had to endure during the first weeks. When one is playing the newbie, times abound when one depends on other people’s feedback. And Debian is made of volunteers, who have a life outside it. Every time I asked, Louis-Philippe was there. I wouldn’t have gotten here if it wasn’t for him. So, thank you *so* much for all that, Louis-Philippe!

Final Words

I would like to close this report with a reflection.

I have been using Debian for many, many years now, and I had been looking for a way to contribute back to the project for some time already. I even did some work on a non-packaging Debian project. That being said, I never managed to deliver much, really.

So, the very existence of outreach programs as this one is, in my humble opinion, crucial. In my case, the funding I got through the GSoC program was instrumental in being able to allocate time for this endeavor, and to finally get started contributing to Debian. Plus, it has had a very positive impact on me; in many ways, some of which I am only starting to discover now that the project is ending.

When I put things into perspective, this project is very important for me. Actually, it is nothing but the first step within a long-term journey: becoming a Debian member. Hopefully, I would like to be able to apply for Debian membership by the end of this year.

Questions?

Thank you very much for your time reading this! I look forward to hearing (or reading) your feedback. Please come and meet with the Debian Clojure Team Moreover, I will be in the Clojure BoF on DebConf2021. Moreover, do not hesitate to send me an email.

Data

Task Status

  • Required Tasks:

    • T1: Setting up a full Debian packaging development environment and learning the basics of Debian packaging.

      • Successfully completed the first part during the Application period.

      • Successfully completed the second part during the Coding periods.

    • T2: Identifying and packaging the missing dependencies to package clojure-cli.

      • Successfully completed as of the end of Coding II.

    • T3: Packaging clojure-cli.

      • 90% done as of the end of Coding II.

    • T4: Updating clojure to use clojure-cli.

      • To be completed after GSoC.

    • T5: Updating the Clojure Packaging Guide with information on how to use the new clojure-cli scripts.

      • Improved existing documentation. To be completed after GSoC.

  • Optional Tasks:

    • T6: Writing Lintian tags to make Clojure packaging in Debian more robust.

      • To be completed after GSoC.

    • T7: Working to automate Clojure unit testing in autopkgtests using autodep8.

      • To be completed after GSoC.

    • T8: Updating older Clojure packages not built using leiningen or clojure-cli.

      • To be completed after GSoC.

Packages

  1. Updated packages:

  2. New packages:

    • tools-gitlibs-clojure -- Clojure API for programatically accessing git libraries. ITP: #905543 in NEW.

    • “ITP: tools-deps-alpha-clojure -- functional API for dependency management and classpath creation” https://bugs.debian.org/891136 Needs to be uploaded by Louis-Philippe.

  3. In-Progress packages:

    • “ITP: clojure-cli -- upstream CLI entrypoints for Clojure” https://bugs.debian.org/891141 90% done - Package completed. I only need to finish implementing the transition from existing ‘clojure‘ scripts. To be completed after GSoC.

  4. Opened ITPs:

  5. Reported bugs

Other

  1. Interactions with upstream in the Debian-Clojure mailing list:

    • Many productive interactions with one of the upstream developers, Alex Miller (June, August).

  2. Wiki page:

    1. https://wiki.debian.org/Clojure/Goals/ClojureCLIToolsInDebian
Tags: debian clojure gsoc