Intro

My name is Leandro Doctors (“allentiak” on IRC), and I’ve been the GSoC intern working with the Debian Clojure Team during 2021. This is my final report. You can also check my original proposal and my first report.

Summary

Whereas the raw data may not sound by itself very positive, my personal conclusion is. This is, whereas I didn’t fully finish the required deliverables envisioned in my original proposal, I do feel I am much closer to, eventually, becoming a Debian Developer. So, by all means, I consider this project has had a positive outcome.

Project

The goal of the “Clojure Build Tools in Debian” project was to provide Clojure Debian users with some of the latest advanced build tools and libraries the upstream Clojure developers have been lately working on. These include tools.deps.alpha, a library for dependency graph resolution and classpath building, and the CLI tool clj, for REPL interaction. If time permitted, I was also to improve the quality of both new and existing Clojure packages, and the overall Debian Clojure packaging process. My mentor was Louis-Louis-Philippe Véronneau, and my co-mentor was Utkarsh Gupta.

Motivation

Why this project? On the one side, if you’re a Clojure lover like me, you may have noticed that the Clojure experience in Debian is, as of mid 2021, well... still quiet limited. Additionally, this project aligned with my own background in Free Software community building and my research interest in Peer Production.

When I mention how limited today’s Clojure experience in Debian is, I can see two reasons for this, deeply intertwined. The first one is that there currently aren’t many Clojure-specific packaging tools in Debian (such as a clojure-debian-helper). The second reason for which we only currently have a suboptimal Clojure experience in Debian, and probably the root of the previous one, is that many core build tools and libraries for the language have not simply been packaged yet. My project aimed to attack that seemingly root cause.

As I said, another reason for me choosing this project is my own experience as the Co-founder and Leader of, probably, the first Free Software Community experience in my hometown of San Juan, Argentina. That interest in Free Software evolved in a first PhD attempt in what is now known as the field of Peer Production. A subject that has lived within me as a research interest during my day job at a University.

Being a Clojure fan, it felt only logical combining all those interests somehow. And this project seemed like the ideal combination.

The Debian Clojure Team

I’ve been working with a small, yet very warm team. The current incarnation of the Debian Clojure Team exists thanks to the hard work of three people.

Elana Hashman (aka “the Clojure necromancer”), revived the team around three years ago. Later on, the team gained the invaluable presence of Louis-Philippe Véronneau and Utkarsh Gupta (my mentor and co-mentor, respectively).

Together, these “Three Musketeers” have maintained the team alive, allowing us, Debian users, to enjoy Clojure.

Status

During the first part of my project, I mainly worked on learning the basics of Debian packaging, and got my first package uploaded. I have to thank Louis-Philippe, Utkarsh, and Elana for their immense patience and support during that part, as it took me quite some effort grasping the basics of Debian packaging.

During the second part of my project, I worked on my last packages, and almost completed the originally required scope of the project. I only have to finish working on the transition from the currently provided set of packages (based on a Debian-specific clojure runner) to the newly provided upstream clojure and clj runners.

Unfortunately, I didn’t have much time left to start working on the opportunities for improvement already identified by the Debian Clojure Team originally outlined in my proposal. Whereas I did update one older Clojure package not built using leiningen (tools-data-xml-clojure), I didn’t write any Lintian tags to make Clojure packaging in Debian more robust, nor worked towards the automation of Clojure unit tests in autopkgtests via autodep8.

Deliverables: Data vs. Conclusions

If we are to talk about deliverables, we should start with the data. According to my original proposal, I was required to provide both new and updated Clojure packages accepted into Debian “unstable”, and updated Clojure packaging documentation. Additionally, if time permitted, I was to also provide new Clojure Lintian tags merged by the Lintian maintainers, and new Clojure autodep8 scripts merged by the autodep8 maintainers. Whereas I partially accomplished both required tasks, I didn’t manage to start working on any of the optional deliverables.

When looked in isolation, those numbers may look somewhat disappointing for some people. However, I can draw a much more positive conclusion. Why?

Firstly, GSoC is supposed to be a learning experience. Moreover, as I said in my original proposal, “I approach[ed] this project as a great opportunity to, finally, start my journey towards becoming a Debian Developer”. In that sense, I consider the time invested into this project fruitful. In this way, I have learned the basics of packaging, how to interact with the Debian Clojure Team, and and already got my first packages accepted. Plus, I’m looking forward to continuing to work with the Debian Clojure Team so I can attain the original scope of the project. Therefore, all things considered, I can consider this experience as a moderate success.

Lessons Learned

Technically speaking, if I have learned one thing during these weeks, is that packaging, although easy to be underestimated, is by no means a trivial process. As any Debian Developer surely knows, the onboarding process can take some time. Plus, what is easy for some people, can be difficult for others. In my case, this was quite evident. Whereas I can speak several languages and learning new ones takes me little effort, grasping the basics of packaging took me (literally!) blood, sweat and tears. Indeed, the packaging learning curve was quite steep for me.

That being said, I did learn a thing or two about packaging. So, if I managed to get here, I’m sure many others can. It may take them more or less time than what it took me, but learning (at least the basics) of packaging is an achievable goal.

Technical skill learning aside, I value very highly the non-technical skills I have so far improved during this project.

For instance, I also learned that it can take some time to adapt to real-time online communication. Before this project, remote working meant either exchanging emails or getting into video or audio calls, with a low emphasis on chat-based interaction. Early on, I realized that the Debian Clojure Team interacts almost exclusively via, well... chat! And those two approaches are very different indeed. It has taken me some time to adjust, but I’ve improved greatly in this aspect as well.

Finally, improving my time management skills has been also a key part of this process. Whereas I had already been working remotely for over a year and a half already, my day job is not so interaction-dependent as this project (specially in the beginning). So it took me some time to adapt to this way of working, and to plan my workload so I could use those waiting moments to advance in other parts of the project. Still a lot to improve here, but improving nevertheless.

Acknowledgments

I first have to thank upstream. More specifically, one of the upstream developers of the clojure-tools, Alex Miller. Everytime I needed specific information on what do specific parts of the Clojure CLI tools’s codebase do, tools.deps.alpha do, he popped up a reply in a matter of hours. He has shown genuine interest in the success of is project during by carefully replying to my emails with detailed explanations of code intent and form, both in private and in public conversations. Thank you for all that, Alex!

Let’s move on to the Debian Clojure Team.

First, Elana. I thank Elana for her initial openness when I first contacted her about this idea. It was *her* who initially contacted Louis-Philippe so he would become my mentor. I wouldn’t have started to work on this project if it wasn’t for her. Plus, she provided quite a piece of advise in more that one ocassion. So, thank you very much for all that, Elana!

I also thank Utkarsh, my co-mentor, for his overall technical advise. And a special mention to his initial help to setup my Matrix client for OFTC chat. At that moment, it was *him* who took the time to help me in real time so I could solve that problem. So, thank you very much for all that, Utkarsh!

I finally have to thank Louis-Philippe, my mentor, for his patient guidance during the whole process. His dedication and hard work has been *instrumental* for my progress. And a special mention for his tolerance with respect to some unforeseen personal circumstances I had to endure during the first weeks. When one is playing the newbie, times abound when one depends on other people’s feedback. And Debian is made of volunteers, who have a life outside it. Every time I asked, Louis-Philippe was there. I wouldn’t have gotten here if it wasn’t for him. So, thank you *so* much for all that, Louis-Philippe!

Final Words

I would like to close this report with a reflection.

I have been using Debian for many, many years now, and I had been looking for a way to contribute back to the project for some time already. I even did some work on a non-packaging Debian project. That being said, I never managed to deliver much, really.

So, the very existence of outreach programs as this one is, in my humble opinion, crucial. In my case, the funding I got through the GSoC program was instrumental in being able to allocate time for this endeavor, and to finally get started contributing to Debian. Plus, it has had a very positive impact on me; in many ways, some of which I am only starting to discover now that the project is ending.

When I put things into perspective, this project is very important for me. Actually, it is nothing but the first step within a long-term journey: becoming a Debian member. Hopefully, I would like to be able to apply for Debian membership by the end of this year.

Questions?

Thank you very much for your time reading this! I look forward to hearing (or reading) your feedback. Please come and meet with the Debian Clojure Team Moreover, I will be in the Clojure BoF on DebConf2021. Moreover, do not hesitate to send me an email.

Data

Task Status

• Required Tasks:

  • T1: Setting up a full Debian packaging development environment and learning the basics of Debian packaging.

    • Successfully completed the first part during the Application period.

    • Successfully completed the second part during the Coding periods.

  • T2: Identifying and packaging the missing dependencies to package clojure-cli.

    • Successfully completed as of the end of Coding II.

  • T3: Packaging clojure-cli.

    • 90% done as of the end of Coding II.

  • T4: Updating clojure to use clojure-cli.

    • To be completed after GSoC.

  • T5: Updating the Clojure Packaging Guide with information on how to use the new clojure-cli scripts.

    • Improved existing documentation. To be completed after GSoC.

• Optional Tasks:

  • T6: Writing Lintian tags to make Clojure packaging in Debian more robust.

    • To be completed after GSoC.

  • T7: Working to automate Clojure unit testing in autopkgtests using autodep8.

    • To be completed after GSoC.

  • T8: Updating older Clojure packages not built using leiningen or clojure-cli.

    • To be completed after GSoC.

Packages

1. Updated packages:

   • tools-data-xml-clojure https://tracker.debian.org/news/1244465/accepted-data-xml-clojure-020alpha6-1-source-into-experimental/ Stuck in experimental until the latest version of Leiningen is packaged.

2. New packages:

   • tools-gitlibs-clojure -- Clojure API for programatically accessing git libraries. ITP: #905543 in NEW.

   • “ITP: tools-deps-alpha-clojure -- functional API for dependency management and classpath creation” https://bugs.debian.org/891136 Needs to be uploaded by Louis-Philippe.

3. In-Progress packages:

   • “ITP: clojure-cli -- upstream CLI entrypoints for Clojure” https://bugs.debian.org/891141 90% done - Package completed. I only need to finish implementing the transition from existing ‘clojure‘ scripts. To be completed after GSoC.

4. Opened ITPs:

   • “ITP: libtools-build-clojure -- a library for building artifacts” https://bugs.debian.org/992737

     • To be completed after GSoC.

5. Reported bugs

   • “libcore-specs-alpha-clojure: should provide (symlinks to) .jars in /usr/share/java” https://bugs.debian.org/992735

   • “libspec-alpha-clojure: should provide (symlinks to) .jars in /usr/share/java” https://bugs.debian.org/992736

Other

1. Interactions with upstream in the Debian-Clojure mailing list:

   • Many productive interactions with one of the upstream developers, Alex Miller (June, August).

2. Wiki page:

   1. https://wiki.debian.org/Clojure/Goals/ClojureCLIToolsInDebian

Hi, everybody!

For those who don't know me, my name is Leandro Doctors (allentiak on IRC), and I'm the Debian Clojure Team's GSoC 2021 intern. My mentor is Louis-Philippe Véronneau[*] (pollo on IRC). My co-mentor is Utkarsh Gupta (utkarsh2102 on IRC). My 'no-mentor' :) is Elana Hashman (ehashman on IRC).

In this message, I will summarize what I've been up to during the Coding I period (June 7th - July 16th).

TL;DR: my updated data-xml-clojure package was accepted[1] in experimental on Wednesday night :-)

[1] https://tracker.debian.org/news/1244465/accepted-data-xml-clojure-020alpha6-1-source-into-experimental/

Now, let's tell the full story.

During the first days of the "Coding I" phase, I did some research on the clj dependencies. And after the precious feedback from Louis-Philippe, Elana, Utkarsh, and Alex Miller (the upstream developer of clj, among many other libraries), I came up with the following packaging strategy.

1. update org.clojure:data.xml (data-xml-clojure) to 0.2.0-beta6.
2. package org.clojure:tools.gitlibs (tools-gitlibs-clojure).
3. consider updating libjsch-agent-proxy-java, jgit, libtools-cli-clojure (all of them already in Debian).
4. consider packaging com.cognitec.aws:* packages.

According to my original proposal, I should have completed all four tasks during Coding I. Looking back, the main lesson from these past weeks is a known classic: my timeline was too optimistic: I definitely underestimated the difficulty of the packaging process. Out of the four tasks, I only finished the first one.

There were many challenges I had to overcome in order to update the library from version 0.0.8 to 0.2.0-alpha6:

This what I did:

First, I patched the source code so:

• • we avoid needlessly packaging data-codec-clojure (not currently in Debian).
• • we can ignore Clojure-Script-related dependencies.

Then, I completely overhauled the packaging code (this is, what goes inside debian/).

• • added support for automatically tracking newer releases.
• • the package now builds with Leiningen, instead of plain Java.
• • it now actually supports autopkgtest (the existing test was trivial).

All this improved the quality of the package.

I also improved the Clojure Packaging Tutorial[2] to make the process easier to follow.

[2] https://wiki.debian.org/Clojure/PackagingTutorial

Looking back, it is almost as if I had started packaging the library from scratch...

But, more that what I produced, I think the most important part of all this is what I learned during these weeks.

As it was the first time I ever packaged anything in Debian, I had to learn the basics, bump by bump. (And, oh my, I surely did bump quite a few times!)

• • Basic packaging workflow.
• • Setup sbuild[3] to make it work with gbp[4] (when I had rebuilt bidi-clojure from scratch, I had used plain sbuild.)
• • Learn how to patch upstream files with quilt[5] (actually, via the dquilt frontend). This was the first really difficult task to do, since it took me quite some time to grasp it. Looking back, I think I was simply scared of breaking something :)
• • Understanding debian/ files.
• • Understanding debian/tests (autopkgtest) files. This was particularly difficult, since the doc wasn't clear about it. So I then improved the Clojure Packaging Tutorial[2] to make the process easier to follow.

[2] https://wiki.debian.org/Clojure/PackagingTutorial

[3] https://wiki.debian.org/sbuild

[4] https://manpages.debian.org/unstable/git-buildpackage/gbp.1.en.html

[5] https://wiki.debian.org/UsingQuilt

Definitely, all this was worth it. After all, my updated data-xml-clojure package was accepted[1] in experimental on Wednesday :-)

[1] https://tracker.debian.org/news/1244465/accepted-data-xml-clojure-020alpha6-1-source-into-experimental/

Now that I've learned the basics of packaging bumped enough to get my package accepted, I'm hopeful I can ramp up and catch up with (at least most of) my original schedule during Coding II.

tools-gitlibs-clojure, you're next :-)

Thank you very much Louis-Philippe, Elana, and Utkarsh (plus a special mention to Alex) for your precious support during the last few weeks!