Of these, the one I want to talk about here is the switch from an imperative to a functional programming paradigm. All my previous work had been in imperative languages, from C through C++ and C#, with side-trips into Java along the way. The core services at CognitiveScale are developed using Scala as our language of choice, and with a very functional style.
Scala is something of a hybrid, which bridges the OO and functional worlds, and probably the main reasons it was chosen were:
- Flexibility in style - it can be used in a variety of ways, from simply 'a better Java', through to a much more Haskell-like pure functional style. This allows for quick on-boarding of Java programmers (or those with similar language backgrounds, such as C#) while also supporting much more abstract styles, with the benefits accrued from the functional world (as I'll discuss later). The choice can be made by individual teams, or components as appropriate to them.
- Brevity and expressiveness - very little boilerplate is required, and a lot can be achieved in a small number of lines of code
- Good community - Scala is a sufficiently popular language to have a good online community that seems to be generally friendly and open.
- JVM ecosystem - because Scala is a JVM language, with strong Java inter-operability, there is a vast resource of open-source libraries available for almost any purpose you can imagine. Notable open-source building blocks in use at CognitiveScale include:
- Apache Kafka for asynchronous queuing
- Argonaut for JSON serialization
- Zookeeper as a foundation for distributed cache coordination
- HTTP4S as the HTTP layer for REST services and clients
- ScalaZ, of which more later, which is a categorical functional library for Scala
- Strong type system. This is perhaps a personal choice and there is room to differ on the pros and cons of strong vs weak typing in a programming language. However, the group of engineers we have at CognitiveScale would all fall somewhere on the stronger-typing end of the spectrum (some more extremely so than others!)
Within the core development team at CognitiveScale, we hew very much to the functional end of the style spectrum, strongly influenced by trends that largely began in the Haskell community.
For me, as a new developer at CognitiveScale (almost a year ago now), this was a huge paradigm shift, and it has an admittedly steep learning curve. For someone with an imperative background, the idea of writing complex code that eschews mutable data structures ([very] loosely speaking, no modifiable variables) and doesn't permit side effecting functions/methods (the ONLY thing a function does is return a value), is initially mind-boggling!
Fortunately the general atmosphere at CognitiveScale is one that promotes openness and continued learning, and one manifestation of this is a number of ad-hoc skill-sharing groups on various subjects, with semi-regular presentation and teaching sessions. One of these teaches functional Scala (another interesting one is a series of semi-regular machine-learning talks from some of our machine-learning experts). In the Scala case, we are fortunate to have on the staff a senior engineer with deep experience of the pure functional style, from both the Scala and Haskell worlds, who is also motivated to promote functional techniques and help develop deeper skills in the wider engineering organization.
My experience has been that it takes a while to get up to speed on such a different paradigm, but that once a certain threshold is reached (which was probably about 3 months in for me, though I'm still learning now), significant gains become apparent. These manifest in a few ways:
- Often pieces of code seem to 'write themselves' - so much compile-time checking arises from the combination of strong typing, and powerful parametrically typed building-block abstractions, that there can be very limited choice in the details of implementation once you know what the types involved are, and what it's going to be assembled from. This mostly occurs at the level of individual functions, but the overall experience is almost like 'discovering' the solution as you follow a chain of compositional reasoning, rather than having to invent it from a blank slate.
- Things frequently 'just work' - the experience of (at least unit) tests that just sail through the first time you run them becomes much more common than I have known it to be in previous environments
- Reasoning becomes local rather than involving factors of more global context. This keeps the 'cognitive frame' small and easy to hold in your head. Code becomes much more understandable in smaller pieces. This also promotes reuse through composition.
- Concurrency is much less error-prone and easier to reason about, because coupling via side effects is not present
- Productivity and velocity increase - once you learn how to leverage higher-level abstractions, and to reason about the constraints and laws they obey, complex operations turn into relatively simple compositions of pre-existing building blocks. This is enabled by the functional nature of the language, but amplified considerably by the adoption of a pure-functional style and the categorical framework that ScalaZ provides us.
Overall, programming and debugging start to feel more like algebraic proof (which suits me since I'm a mathematician by educational background), and rigorous reasoning about the code is possible, at least locally. Getting up to speed on this approach to developing software is a considerable investment, but one that has a big pay off. Working for a company that is prepared to make investments in its employees like this, and give them the space for continued development, is something that makes going in to work each day something to look forward to.
I would therefore recommend any serious software development organization, which values their employees as long-term investments, to explore this approach to creating software. Try it on a relatively isolated project with a small group of developers who are keen to explore new techniques first, and let them promote the results more widely as they learn to leverage the benefits.
It's paying off in big ways for CognitiveScale, and for those of us that work there as individuals.
I'm sure it can work in many other organizations also if they are prepared to try.
