The Dark side of being a Software Developer

IBM launched a campaign almost five years ago : „Where teams are heroes“. On conferences they distributed light blue t-shirts on whose back a team of software developers was pictured as super-hero cartoon heroes. We all want to be such heroes don ‘t we? Be famous like Kent Beck, Martin Fowler or Uncle Bob Martin , Anders Hejlsberg, Scott Hanselman and others, don’t we?

And we all kinda live and work in that dream. The long hours and efforts spent on last ditch deadlines, the lengthy and emotional discussions about a problem and our effort to be just the man who saves the team and the project. We all like to be the person people go to fix hard problems.

I lived the dream also. I worked day and night, fifteen hours a day weekends and holidays on tough projects which needed to be finished by a certain deadline or the client and our company will be in grave peril threatening their very existence. Does it sound familiar? I even worked several times from dusk till down, twenty four or even thirty hours consecutive too meet a deadline and afterwards I felt like a god. I’ve leaded a team to hell and beck and completed those damned screens just in time for the client presentation in the morning. And I’ve been doing such shenanigans for some time now.

What is the cost of such a way of life? What is the price we as people pay for doing such a service for our company and our clients?

What is the dark side of being a software developer?

SAS Special Forces WWII (Africa)

Let me tell you my story. Something that happened to me at the turn of the millennium.
  Let me tell you what price I paid for my way of life.

My wife threatened to divorce me a dozen of times, threw me out of my bedroom (temporarily thank God) , yelled at me continuously and generally made my life a living hell. Often I was made to choose between my family or my work. And both seemed critical at that moment. And I really wanted her to understand that what I was doing was important and critical and serious consequences will happened for multiple people if I don’t deliver. And she would yell at me and tell me that I’m not a soldier, or a doctor or a policeman that I do not work 24h a day and I’m not on back and call every day of my life. The situation was more drastic when my son or her were sick and I had to work.

Ultimately I tried to work from 5 a.m to 17 p.m , but the stress and the long hours and the lack of sleep just put me into bed for a couple of weeks. What a bummer.

Not my health, not my wife were the most horrible, most darkest and vilest price I had to pay. It was my four year old son.

One day he just wouldn’t budge from me after lunch. He cried and clinged to my leg and didn’t want me to leave. And I didn’t, but started to play with him. During the play he said to me that he had a dream : “Daddy, I dreamed that next time you will go to your work I will not see you for a long, long time”.

My work gave my son nightmares that he is going to lose me.

When my mother put him to bed one night he asked to her if I’m going to come to bed next him when I stop working for the night and why is daddy working during the night after working the morning and the day and the last five days like that.

My soul froze and my eyes started crying and my scream stopped in my mouth and I cursed my self for being such a monster to my family.

Is it really all that important?

The client set a deadline but made thirty percent change on its original request all which was accepted by us and the deadline slipped day by day into oblivion and often we would rewrite the same feature or screen several times. The work would just pile on with an even increasing defect rate due to long hours and stress and the dead line would remain the same and my colleagues and friend and I would soldier on trying to fix that which was already broken.

During that time I told my self that in the end it will all be worth, everything will pay out and my family will live better and all the people around me told me that I’m being stupid and an idiot.

But I stopped for a minute and made a mental calculation. It just didn’t pay out. The amount of personal loss compared to the business gain I expected and received where just not comparable.

Yeah sure the business is important. It puts food on our table and we must live by our word to deliver something to certain day. I believe it still and live it still. But you know, being a software engineer is just part of my life the other is my family. I still work in the evenings, but not all evenings . I work because I like it, and enjoy it and I get to learn something , not because someone made me. But my place and my obligations are to my family also and that is an area I really, really messed out. 

Stuff I learned working as a scrum master

 

Working as a scrum master I learned a couple of hard truths :

• People have a lot of misconceptions about what a scrum master should be doing
• Sometimes they are true (which makes things worse)

Now If you take a look at the Scrum guide the scrum master has a lots on his plate:

Now the things I most often hear is that the scrum master should:

• make sure there are burndown charts (as the single most important thing a scrum master does)
• organize and run the scrum events (although the guide says that the scrum master should be doing that as needed or as requested)
• remove impediments

I’ll be honest, in relation to burndown charts I never drew one (there are tools for that). I find that tiring and I think there are better uses of my time. But surprisingly this is what I most commonly hear from developers – “My previous scrum master was spending most of his time drawing charts and organizing meetings …..” – what?

Scrum events are important but the thing is that the value of a scrum event is not provided by a scrum master but by the people joining the event. I think that helping people to prepare for the scrum event is much more important then actually running the event it self. I rather focus my self on facilitation then on meeting organization.

Removing impediments its a very important thing. The team has bring business value by working on the sprint backlog as agreed with the product owner. Anything that is preventing the team to do that must be ruthlessly removed.

Now here lies the catch. That could be a wide variety of things. And is really contextual – what is the current situation within the team and the company dictates what is an impediment and what is a possibility to grow.

In situation A the team needs to build a MVP (Minimal Viable Product) to put on the market as quickly as possible. Anything preventing the team to the most important tasks to achieve that is an impediment ( I try to simplify things as much as I can). Now in that situation building a continuous integration server , something the team has no experience with , or making sure all resources on Azure are created as they should or taking care of 3rd line support calls coming to the team or helping write all the unit tests or doing some research or … the list could grow. And all those things could also be things the team could be doing and taking them as learning and growing opportunities.

Its highly contextual.

Now for me the rule of the thumb is as follows:

• Is it part of the teams core skill set – yes (let them do it)
• If not is it needed to achieve sprint goal – yes (ok deal wit it somehow)
• Is this the right time to grow the team on this skill – yes (let them do it, and help them)
• If not take care of it your self or find somebody who will

DotNet Core : The good, the ugly and the just started

I just started out playing with .NET Core on my own time. Getting the feeling for the technology, trying to learn the basic concepts and getting some basic programming tasks done.

Wow.

Working in .NET for the last eight or so years I really find this one much more different then the usual flavor provided by Microsoft.

Lets be honest, on most .NET projects I worked the MacDonalds style of programming, where 80% of the problems are solved easily by spending 20% of your project time while you spend the remaining 80% dealing with the remaining 20% of the problems which incidentally are what the the project is about. Go figure.

With this first dotnet core project I spend my time dealing with something completely different. It actually reminds me of the time I was learning to code in PHP and Python (it was years back during my Linux times).

The first thing that struck me was complicated level of setting up a project (project.json configuration) and linking all projects in a solution together and then managing all the different execution and deployment environments. Whaat?

But hell, its fun to deal with these type of issues.

Most of the help I got didn’t come from the actual documentation. It was mostly useless, so I just relied on googling for solutions or going directly to the dotnet core repository and looking trough the source code.

The second biggest problem was that there was not a solution wide way to build all projects and run all tests. Well, that was easily solved by using the custom Task functionality of VSCode.

Something like this:

.vscode/tasks.json

And then in the root of the solution writing this small batch script:
build.cmd

And then with a simple, and familiar CTRL+SHIFT+B , voila:

it builds everything and it even runs all the tests.

The command line interface is simple and easy to understand, and once you get past the project.json madness you will have a literally very powerful platform to work on.

The coolest thing it is that the project is completely open source so well if you find a problem or have a need, the solution is several hours and a pull request away.

Of course I don't see .NET teams jumping all over it right now. I haven't yet tried developing in .NET Core with Visual Studio but I guess as with every version 1.0 of a Microsoft Technology the tooling will suck long time (usually we have to wait till version 3.x till we get the maturity level high enough to develop normally). But hell, I could be very wrong.

Still its a cool thing to play with. If you are curious what I did so far please look at my GitHub profile:

Being a software developer without a University degree

I'm a software developer without a university degree. My family couldn't afford to keep me at the university past the second year and to my misfortune I was not ready to strike out at my own.

I returned to my home town and started working in my fathers small company. It was a three person team. I was working there since I was thirteen, on and off during every school break. Now it was permanent. Or so I thought.

I always wanted to become a software engineer. Since I was a little kid and my father smuggled a Commodore 64 from the Netherlands into Yugoslavia. My mother, who spoke a little German, helped me to learn BASIC programming at the age of nine from a Dutch manual. By the age of eleven I was writing a combination of assembly and BASIC and writing my own, albeit simple arcade games.

If I could not get a proper education I decided to get one by myself. Luckily for me I was working in an industry that is fully internet driven, and all the resources I needed to educate myself were available online.

I bought a book and taught myself PHP. I wrote a couple of in house projects for my family and after a year I got my first programming job. Well the rest is history. I ended up having a good career so far. It had its ups and downs but so far I managed to keep myself employed and my family provided for.

Not having a university degree in an IT related field is one of my biggest regrets in life.

When searching for new work some of the positions I applied to would not include me in the phone screen part of the interview process, some would never respond to my application.

Regardless of not having a university degree I managed to build a career in software engineering.

I'm proud of my achievements. I self educated myself. It was hard and I still work on it.

How did I do it?

• Find out what are the core skills a software engineer needs to have and train your self in them
• Spend time daily (ideally one hour or more) focusing your self on the core skills
• Read, listen to IT related media.
• Rinse and repeat

I think for the core elements of software engineering there is no better alternative then to buy and read books about the topic. Something which is considered core doesn't change that often and if it does skills learned can be easily be transferred to the new version. Alternatively reading documentation (for example PHP always had excellent online documentation), watching presentations and tutorials (InfoQ is my favorite source about anything software engineering). 

I think if you want to be an excellent software engineer you cannot get away from spending one or more hours a day improving your skills, be it programming, reading or listening. You have to do it. Your job will provide some but not all the skills you need to advance. 

Yes, you will be tired. So I am, in the evening, but not in the morning. So I getup earlier and spend time learning while I eat my breakfast. I listen to podcasts on my way to work, read books in the tram or when I'm waiting for something. 

Chances are you will always find gaps of time you can fill with some learning aspects.

Sitting in front of a computer and coding is critical. There is no alternative, the only advice I can give is sit down and do it regular. Plan for it and do it. Nothing worthwhile doesn't come easy. Being a good software engineer is sometimes a hard life, even if you love doing it as I do. 

I love listening to podcasts, .NET rocks and Software Engineering Radio are my favorite. Years ago when I was working in Croatia I was burning CDs with downloaded podcasts and listening to them on my way to and back to work. It was the best of times. 

Nowadays I use my Safari Online subscription and read books while I trudge in a tram. 

The most important thing

Be positive and keep striving. Life and the world will keep building walls around you, there is no need to build them yourself. You will hit obstacles and dead roads. But I firmly believe if you keep working and doing the hard things you will achieve what you need to achieve. 

The only constant is change

Software development is not an exact science.

Let me repeat that.

Software development is not an exact science.

During my career I've never had the same team, the same project and the same business context twice. Although I learned from all those experiences, and I've extracted principles and rules of thumb each new company, team and project is a completely new experience.

In the heart of all modern software development processes lies the feedback loop.

Try something. See how it goes. Change based on the feedback.  The hear of software engineering lies in the continuous usage of the feedback loop on all level and dimensions of a software project, from the individual coder, from the dynamics in the team, from the project management standpoint and from the interaction with the client.

When an engineer understands this simple fact all the rules, all the processes pale to the simple fact that their are all just tools used in the service of maximizing the gains from the simplest principle of the feedback loop.

Governing a software project there are multiple forces. The team that builds it, the business that sells it and the customers that buy/use it. The huge mental strain that goes into building a software product and its plasticity render it completely impossible to fathom at the begging, to set it hard and simple to build as a hause that can be designed by a single person then rebuild across multiple locations by less skilled people. But on the other side its plasticity render it hugely flexible, and prone to changes and adaptation.

A software product is never fixed. The people who build it are never constant. The people who want it are never constant and the constant the product operates in is never constant. Accepting that the only state a software team exists is a state of fluctuating chaos a consolamce of sort can be found that are never right answers and that the only truth is a process of continuous improvement and adaptability.

Change is the only constant in life.

Change is pain.

Change is growth.

To survive one must accept constant change.

Are integrated collaboration solutions the future of the internet?

I hate signing up for services on-line. Just going trough the process of typing my username, email address, password, personal data over and over for all the services I would like to use makes me crazy.
Luckiliy with the the advent of oauth and other federation services nowadays I can signup form most services using my Google or Facebook account (the two I generally use mostly). One-click signup made my life easier.
A nice feature of using web services is the ability to reuse data, content and features from one service (e.g. Google Drive) within Another Service provider (Draw.io) and have changes to my work published to a group of people using a collaboration tool (Slack).
Most online service providers expose REST endpoints for almost every feature they provide. Some are notification features (reading statuses of artifacts) while other expose behaviors of the service (e.g. posting a facebook status or saving to my google drive folder or publishing a note on evernote).
Are we living now in the world of the programmable web where REST apis replace the old days of bash shell scripting in the unix command line for hackers, where a basic level of programming skills enables us to automate and integrate our online work life?
Tools like IFTTT  or Zapier provide simple to use integration commandlets between various services which is a nice start for that entire space. But looking at their API's they are really simple. Comparing them to all the various possible REST apis published by almost everyone the amount integration and collaboration possibilities are enormous.
Leveraging various cloud based solutions its trivially simple to set up a continuously running background service in the cloud which executes various types of checks and integration tasks on a regular basis. I would recon that for lean starups and small organizations one of the first automation priorities they need to execute is to automate all their online services and collaboration tools.
New organizations are not just bound by one service provider but utilize a wide array of services offered on the internet, all of which can either be used for free or very cheap. Adding users and connecting all those services in one go is where most organizations will leverage the ROI of utilizing all those solutions.

Applying the Bosnian Model to the European Union

Bosnia and Herzegovina is a broken country. No questions about it. It was broken by the fall of Yugoslavia, but bearing the brunt of the Yugoslav wars and now by the separation into national entities. Its a place where latent conflict still lurks and which hangs together by a thin thread of heavy bureaucracy and legislature which is so prevalent in many lives of BIH citizens.

But there a silver lining. Bosnia is a micro representation of the European Union as a whole, a real life testing bed for European ideas and models. The EU has long and bloody history of conflicts,ethnic cleansing and old grudges popping up after five minutes of talking to a national of another EU country.

With so many languages and cultures living in the same place going about and handling business across the EU its not that easy. Especially the language issue. Lets face it , English is the lingua franca of the world and if you want to hop about the EU the only language you really need is English. It helps to know the local language but how can you expect a single person to be fluid in all the twenty something national languages of the union?

In Bosnia there are three main languages :

• Croatian
• Serbian
• Bosnian

with a small caveat that those three languages are virtually identical. If you are a speaker of any of those three languages you can go and communicate with speakers of the other two languages without much issues. The grammar is mostly the same and the vocabulary is virtually 80% identical. But if I was required to write and speak perfect Serbian or Bosnian I wouldn't be able to. 

The solution is multilingualism. All three languages are equally official.In practice you can get official papers and fill official forms using any script (Cyrillic or Latin) and language of choice. The only caveat being that you need to stick to your chosen form.

On the EU level its a nightmare to work with the local government of the country who's language you don't know. In general you have to provide translations for everything. Its costs money, and its time consuming.

The solution for the Union would be to supplement the local language with English as the official one. You can seamlessly fill forms, sign documents and communicate with either the official local languages or using English.

As Europeans we all still to national oriented, we keep our cultures close and take it to personally anything that could endanger our perceived cultural values. But in order for the union to work we need to transcend our cultural boundaries, egos and national values and be programmatic about how the union works. Our system needs to enable the seamless protocol of ideas, people and business without introducing to much administrative, legislative and procedural overhead. Its bloody difficult just by it self to work on such a scale. We really don't need to make it more difficult.

The layered architecture is broken

In the begging there was the command line , and before that the punching card, and before that we wired steam, brass, copper and electricity into simple calculating machines. Finally we wanted images to move and interact with us so the word brought us the Graphical user interface and we rejoiced in the ability to click, touch, rotate and drag things on our screens.

The systems we brought into the world talked with the operating system using POSIX commands, with each other by sending messages over pipes, tcp channels and finally using the HTTP as the universal communication layer.

The world of modern application development is a world shaped by two opposing communication forces:

• our systems need to talk and have a meaningful relationship with humans
• our systems need to talk and have a meaningful relationship with each other.

What lays in the between is the void we call our software, our business layer, our domains. Our work.

Why do spend so much time talking, writing, ranting, discussing, arguing about everything elese except how to best to build our main deliverable, the core of our application systems?

The reason is simple. Communication is hard. Communication is the biggest problem we humans have not yet successfully solved. 

As human beings we are tough the basics of communication, the rules governing our interactions, the principles of speaking and writing and scant else. We are left to our divise to learn the best way to communicate with each other in a variety of different context.

Software is built to simplify human systems. To facilitate interactions, processes and in the end to ease the way we interact with each other.

A modern application is not longer a calculating, mathematical system. Its a living, breathing communication entity which needs to properly interact with humans and other digital systems. It needs to be able to talk, to understand and communicate.

How we built our systems is all centered how those systems need to communicate with all other interested parties. The main architectural and development styles are all centered how to most effectively connect our main working deliverable , our logical part, with the outside world.

Nowadays its more easier to find content related how to build UI systems, application services  or persistent storage communication than to find content related how to build the core of our system. 

We all know to how develop and how to solve problems. 

We do not know the perfect way to communicate.

As an industry we are always trying to change the state of the art of  how we develop our software. In the mid twothousands we tried to escape spaghetti code applications where the application logic, external communication points (e.g. what our applications are exposing to the outside - UI, web services etc) and our internal communication points (database, web services we consume or other stuff our logical part needs to work perfectly) and thus we focused on the layer application architecture, the MVC design pattern.

A typical application architecture from the beginning of the millennium looked like this:

I present you the archetype layered application architecture. Our presentation layer communicated with our logical/domain layer who in turn used the Data access layer. On the sides we had our framework and other cross cutting aspects which pervasively transcended our layers.

It was a good model and a good starting point for all our applications then and nowadays. The problem was that it didn't work for all the scenarios we tried to do and in the end it was broken since we couldn't use this architecture to solve our problems. Thus the architecture of our solutions started to dilute, to be broken because it was really hard to stick to this process. We knew it didn't make sense for what we tried to do. 

Alistair Cockburn came with the  Hexagonal architectural model (or the "Ports and adapters") to better describe the architecture best suited for modern application (systems) development. The general idea is more aligned with hour our system ended out to be when have broken out of the layered application design. 

In the hexagonal architectural model a single solution is composed of the following component types:

• One pure logical model
• One or more external communication points (ports)
• Zero or more internal communication point (ports)
• One or more integration environments providing adapters for one or more external communication points and all internal communication points

The pure logical model is the working deliverable our system produces. It has the minimal amount (ideally zero) direct outside system dependencies needed in order to properly function. 

External communication points (ports) are the interfaces, the services, the information offering it provided to the world at large. In order to interact with the pure logical model one must go trough them. A pure logical model needs at least one external communication point, but it should provide more points grouped into different context based on the type of interaction and specif usage requirements of the outside components making use of the pure logical model.

Since external communication points are bundled with the logical model they are allowed to call and make use of each other , or better yet make use of a shared internal external communication point generalizing common outside communication patterns.

Internal communication points (ports) are required by the pure logical model in order for it to successfully complete its work. They are not required, it is possible that the pure logical model does not require services provided by other logical models, third party systems or persistent storage points. 

Internal communication points can called only from the logical model or by each other.

The combination of a pure logical model, its external and internal communication points is the first unit of reusability - a self contained unit.

In order to complete our system we need an integration environment. The integration environment is responsible for taking one or more self contained units and :

• exposing one or more external points (ports) to different clients trough its adapters
• providing adapters  for all internal communication points.

A single self contained unit can be exposed trough different integration points, the specifics of each integration point are not longer a core architectural choice but a plugin, something which can be changed and replaced. Thus the core deliverable of a hexagonal architecture is a specific integration enviroment serving a specific scenario or case.

The critical challenge related to the architecture of hexagonal solutions is the choice of integration patterns. And that is something we have been doing with our applications for quite some time. By applying the same principles used to build layered applications we can quite easily and more naturally build haxagonl solutions.

A specific technical example for a natural hexagonal architectural solutions is the Node.js Express web framework. Express is a very simplistic solution. In essence it provides only two things:

• Routing http requests
• Ability to work with Http Request and Responses on a high level

everything else is handled by introducing components called middleware, which include everything from error utilities, json serializes and deserializes, logging and our application code. 

Measuring estimates in uncertain environments - another view on agile methods

I've spent the majority of my career working in the outsourcing business. Outsourcing its an exciting domain if you love technology, learning and are an adrenaline junky. Luckily I'm all three, so I fit right at home.
Outsourcing is all about making the biggest return of investment (oh, well no big surprise there) by reducing as much the development time possible in order to satisfy all the key client requirements.
When approaching any outsourcing project you have the following guarantees:

• You are never going to know everything you need to build the project
• Some of the technology involved will be unfamiliar to you
• You are not going to have enough time.

In order to succeed in outsourcing you need to have the following three characteristic:

• A self measuring and self correcting software development process
• Strong requirement gathering and managing skills
• Will to learn outside of regular working hours

I like to call outsourcing projects , "The big unknowns" or "Chocolate boxes" (like you never know what you are going to get). As you may fathom, knowing how much its going to take to complete a project is something of a critical information. 

In the classical, bad example model, the evil overlord of the project pressurizes the team to give the shortest estimate on the delivery which is then turned into a strong commitment. 

Notice that I'm talking about pressure here. Mark Horseman from Manger Tools talks about three different powers in an organization:

• Role power
• Technical power
• Relationship power

the pressure in the above sentence is related to the abuse of the Role power, e.g, the might which lies on you by virtue of the job you are doing and its bestowed and enforced by the organization. The principal issue with the use of role power in any engineering organization is that you are dealing with intelligent, educated people working on a project which requires constant high order mental exercises day in , day out. 

You generally can't treat software engineers like they are factory workers, or farm workers where their outputs can be reasonably measured and compared and standard benchmark of operations is actually derived the lower level ability to execute a set of manual operations repeatedly. 

On the other hand you really can't let the cats fly out and what may come will come which is also the donwside of working with smart, intelligent and educated people. They tend to stray, a lot if left alone to much. 

Here we are talking about a case where estimates are forced on a team that has a low certainty level of actually delivering on them. In practice I've seen several different outcomes of this behavior:

• Lots of overtime
• Lots of technical dept
• Lower quality of the delivered solution
• Lower maintainability of the code

to give a summary we have tears and pain to spread around. Yes, yes, we all know that and we all have lived trough those moments.

And here come the agile model. The promise of freedom. Of actually delivering software that works where estimates are estimates which are actually handled properly where we are not committing to a whole bunch of work and doing a whole bunch of things at once without heads or tail where we talk to our customers who are actually part of our teams. Hurray.

By now we are all agillists and we all work in agile environments. We all know SCRUM, extreme programming and the Kaizen of Kanban. A burndown chart is not a wierd excel graph type but a living and breathing artistic expression of our ability to draw on a white board. 

From all agile practices I would presume to point one which I find most valuable (yes I know agile practices can not be taken alone, they must be bunched together, but bear with me), the built in control, measurement and correction mechanism provided by standups, sprint scrums, retrospectives and other "lets huddle around and see where we are at moments".

Lets call all those practices with the same name, lets say - PING. A "Ping" is a regular query to gather measurements about the status of the project. The only difference between each Ping implementation is the time and work scope each ping covers. Daily standups cover well one day of work so their scope is smaller, while sprints and retrospectives each cover a bigger time period.

So why is pinging so great? The greatest benefit of pings is the ability to gather real, actionable data about the status of the project without introducing any pollution in the data set. We are actually measuring what is. And when we have done a couple of such measurements we can derive the current speed of the project. And when we change something in our project or development process we can measure how much the speed of delivery is going to be affected by collecting data from our pings.

I always picture a software project like an organic machine, with gooey parts sticking and intricating them selves. The machine moves but with all the gooey parts you actually can't now at the beginning until it started moving what its speed is going to be. So you start it and it goes slow. You change something and then move it again with periodical check ups to see how its going, 

With this analogy I generally think the best approach is to pick a time frame (short one) a chunk of work and start the project. Until we have gone trough a couple of weeks with regular measurements we are not going to be able to effectively gauge the speed of the project and thus correctly estimate its length and our delivery capability.

Nothing new here, its just the old agile promise everybody is talking about. For me agile its not continues integration or test driven development, planning poker or user stories with neatly arranged burndown charts.

Agile in its essence is the acceptance of the intrinsic chaos and fallibility that is software development where each development effort is generally compromised by wild guesses, blind luck and leap of faiths. Agile accepts the chaos that is our industry, and instead of trying to change it, to mold it into well ordered bits it rids the wave and leverages its strength to deliver value to our customers instead of mounds of stinking shame. 

In my experience the best mechanism to gather an measure the speed of team is to regularly monitor it by just querying the regular status without expressing judgement or implying pressure. The results are always going to be 

Microservices architecture models - How to document and think about microservices based architecture models

Today on the First of January 2015, a special day, with a special name , eleven days before my thirty fourth birthday I watched several conference presentations about Micro-services. One by Adrian Cockcroft and the other by Martin Fowler.

For those who don't yet know Adrian Cockcroft is the chief cloud architect for Netflix, a company that revolutionized cloud architecture and in a cause reaction way started up the DevOps movement. Yep, one thing they don't tell you is that when you transition to an architecture based on visualization, stateliness, and computerization of business features you drive up your costs related to the management and development of tools related to your day to day operations. It really puts Dev into Ops. Gone are the days where you could just put and app in production and forget about it on a day to day basis. Nowadays you need twenty something tools just to run and manage everything.

Well, heck, that's not what I wanted to write about today. As you may have seen I've upped up my software engineering writing and profile these days. I've writing about other stuff lately (mainly on this blog Fighting fitness , with my fiancee, about my wonderful journey into self defense and losing weight) .

The first of January started with a well made plan to work on an project I started a couple of weeks ago. Yeah, I didn't. In exchange I've spent the morning trying to find a formal definition of the architecture model Cockcroft presented in his talk at Dockercon.

I couldn't find it, but instead I found a bunch of articles and resources about the current state of micro-services architectures in the wild.

I've been involved in a government project for the last year or so, changing and modernizing a national VAT system IT infrastructure. As part of the this project we build a service oriented (micro-service based) infrastructure with the goal to reduce the load on the main VAT system and ease the development of supportive business functions (e.g. bounded context) like a separate VAT Portal for Taxpayers or a system for VAT refunds for foreign companies. And we all did that with a fairly conservative .NET stack (its the tax authority, we live conservatively).

During his presentation Cockcroft has shown several architectural graphs of the current Netflix architecture. The looked like nineteen seventy fractals from a weed high math graduate from Berkley, all accompanied by a comment that it is impossible to show an architectural model for Micro-service based architectures.

Micro-services based architecture can generally be tough as :

• Application topology (fractals baby, fractals)
• Data processing and transformation system (Event driven, space and time unbounded ETL)
• Application architecture

and yeah if we are talking about the network topology we have software that can discover where, what and why we have something on our network and draw a pretty picture for us to impress everybody else but for all intent and purposes its completely useless. 

But software architecture is not about that. Its about the overreaching model of the system, the pattern we use to think and build our stuff. 

When we are designing a standard application we are never doing detailed class designs. Almost never, we do it only for the most critical parts. Instead we layout the building blocks and discuss how are they going to interact and what patterns are we going to use. 

So, why should micro-services architecture be different. Cockcroft presented a modeling framework how to describe an micro-services architecture in really simple terms. 

On bottom, the deep blue see we have our development. The basic languages (.NET, ASP.NET MVC) and application containers (IIS, Apache Tomcat, Glassfish) we are using to build our stuff. 

Next to it we have the Operations what is our main deployment infrastructure (virtualized servers in our local data center, could services like Amazon, Google App Engine or Microsoft Azure) on top of which we are actually deploying our products.

The Datastore is an interesting thing. Which database technologies we are using and their order of importance (e.g. SQL Server as our main bounded context databases, while we are using SQL Lite for less important we don't care if we delete storage). 

On top we have our micro-services element, the dev and the ops of DevOps:

• Tooling (basic development and deployment tools - think unit testing, packaging, integration)
• Configuration (with lots of servers, databases, services we need lots and lots of configurations)
• Discovery (with things popping up and dieing all around us in a really chaotic fashion we need to figure out how to discover things)
• Routing (back to the chaos of microservices, we really don't want to hardcoded every url and resource - think load balancing)
• Observability (we need to monitor and log what the hack is going with our process , logs and events and process tracing).

Does this model help us build such an architecture. No. But it helps us see which technologies and areas we need to think about, what we have chosen and how to connect everything with everything else. 

We IT people. We developers. We administrators. We engineers, UX experts, Scrum masters and evangelistwe really like our Koolaids. We drink them mouth full and enjoy the bitter test disillusion 

The promise of micro-services is the promises of using a different technology for different projects, freedom from repressive and outdated technology stacks, a better and brighter future of easy to understand code. 

Right. 

Microservices as an architecture pattern are good for somethings. But they are not silver bullet, and with the ability to use a different technology stack for different projects is well and good should we really do it. 

In the end we are going to need people who are going to maintain all of that, and if we don't have specialist for that specific technology we need either to retrain people or build something from scratch. 

I'm a cross platform guy. I'm fluent in .NET, Java, PHP and Python. I've developed software in all those platforms and then some others to my chagrin. But not all people are really happy to learn something new and adapt to a new platform. Most stick to one thing only and live and breathe something they do not need to spend my time to learn about. They can go about their business and develop software with the ye olde technology. 

The modeling diagram above is a sketch of a process oriented architectural framework to for building reactive and scalable systems based on micro-services in .NET (with other technologies as needed). Its generally an intellectual exercise based on the architectures of the systems I've built and designed in the past (and currently working on) and somethings I've been thinking about and reading about.

On the left we start with the users. Living, breathing, stinking, eating, thinking, fidgeting human beings. The human animal with all its drawbacks and none of the benefits of a pure machine system (we miss you so much Skynet).

Our users want to buy stuff, like stuff, talk about stuff and look for suitable life companions. And they use and app or go to a web site have something installed on their machine. Lets say we are fairly conservative and have an application. 

That application has many, many distinctive submodules which all need to be presented to our users in a very unified manner. But we don't wan to develop, ship and version them together since they are fairly independent. Once a user logs in each UI application is fairly separate from the rest. They can also have different scalability request (one is used more than the other).

So we build them as separate applications, with their own infrastructure and deploy and manage separately. In front of that entire farfallini collection is the Application Context Routing App. Its only goal is to provide a single entry point for everything, a basic context and session discovery mechanism and to route requests (and their state) to specific application contexts and return specific responses from each separate business function application.

On the right most section of the system we have our micro-services silos. Out business domain bounded context. Each micro-service deals with specific business area. In general are not talking about on or two services, but a slew of services specializing for a specific context (Users, Accounting etc).

They each have their own data store and may be either called by Business function Applications or other Microservices.

How do we connect all of them? In the most simplest of terms we use the following two technologies:

• Message queue (for eventing)
• Web API Gateway - for routing micro service calls (and serve as a circuit breaker in case of service failure).

The general idea is that we want our system to be automated as much as possible, self discoverable and self healing and as much de coupled as possible.

In order to achieve that our services will:

• Raise "I'm here events" or "I'm dieing Jim" events trought the event system which will be read by the Web API gateway which will in turn manage the service registry (who is where)
• The web api gateway will queue service calls in case of failures (for important operations) so when our services fail we have a log of all calls made to them and we can execute them as soon as the failing service is up again (e.g. circuit breaker)
• Services need to handle data stored in other services (e.g. have their own local copies) so they need to publish their internal events and react to events published by other microservices.

Does anyone think we are back into the layers end. 

1. Web UI (Web aps instead of Controllers)
2. Domain (Microservices instead of Business logic layer)
3. Web api Gatweay, Message Queues and routers as the glue that holds everything together

Its all the same pasta, from the lower layers of computer interactions to our brave new world of microservices and cloud computing. 

When building microservices based architectures in the .NET world we need to really reduce the amount of time and configuration we need to do in order to build such a project. NuGET is all well but setting up a ASP.NET MVC Web API project, loading all dependencies, handle all Web.Conf configurations, create all the infrastructure classes we need is really bother.

Luckily our heroes from the Visual Studio and ALM teams come to our rescue. First of all we are not going to rewrite everything from scratch. We need to standardize whenever is possible, but make that standardization optional (we can ignore it when we need it) - call this convention over coding. We are creating a convetion of how we built things.

So if we have a fairly standard way of building our microservices why not create a set of core libraries we are going to load to ease our development effort. I call them Definitions. Definitions of terms and interactions. 

Each microservice is going to have:

• Entites and other domain goodies
• A way to publish services and connect to other services (we really do not need twenty way to access or read data from an internal web service)
• A way to publish and to react to events
• A way to interact with the Web API gateway
• A way to access the database (or database groups).

Great, we write those libraries and publish them into a local NuGet Package repository. Next we build a sample Microservice project that works. A hello world project, or event project groups. 

And with the greatness that is Visual Studio we create project and item templates. We version those into our source configuration. And whenever we need to create a new project we use such a template.

In the end if we think about service oriented architecture, or application architecture in general its all the same thing just looked from different angles and with different terms to throw around. Each type of system has its own challenges and costs.

The important thing to remember is that systems are built by people and how those poeple think and organize is going to be reflected in the system they've built (Coneways law). 

As a software engineer I've worked on both brown field and green field projects. As a young engineer I thought that green field projects were the best. The freedom to experiment and do what I wanted to learn and use the technologies I needed to learn instead of the mess somebody left for me to work on.

And then I came upon a legacy brown field project. Large in scale and large in years with a rich and complex history behind it. With problems, with issues and design decisions that stuck and made life miserable for developers for years to come.

And I learned.

Oh, boy how I learned.

I would advise any young engineer to work maintenance on any large system for a year or so. The possibility for growth and learning are endless. There is no bigger learning incentive then adversity put upon your not so strong back.

On December 5th I've held a talk for the HTML5 User group in Banja Luka where I worked for a year or so. It was the second such talk. It seems they liked the first so they invited me for one more.

As a theme for the talk I played with the idea of extending the concept of brown field software development to organizations. How to develop and integrate new software into existing organizations, organizations which are not your own which at times may be adverse to you , have their own quirks and rules.

Not to bother any one, the gist of the matter is that there are three key elements, ideas, issues, topics, and agendas one must cater to when coming to work as an outside entity into an existing organization:

• Fear
• Communication
• Realism

Fear is the driving force of human kind. Fear of death, the tax man , of cheating spouses and public humiliation. Fear is to life what a cone is to an ice cream ball. Its every where, its all around us and its the primary motivation for many of our actions. As a consultant coming into a new organization you must realise that many behaviors and rules are fear motivated. Fear of change, job loss, learning, fear of fear it self. 

Communication in brown field organization development is the art of finding and understanding all information exchange pathways, who and where are the key players and what is understood, what needs to be said and what needs to be kept silent. It is imperative to join the mainstream communication pathways as soon as possible. That water cooler is your friend and that bear with the guys is never just a be.

Realism is the main principle which should govern your life. We all wish things were different. But they are not and may never be. Real progressive change is painful difficult and takes continuous effort over time. And comes with small increments. Find out what is really possible and focus on that. Everything else is a shared pipe dream. Chase your dragons somewhere else. 

The emotional health of a software system

 

The eternal truth to software development is its inability to produce any meaningful systems without a coordinated effort from a diverse team of individuals, lots of disparate activities ranging from design to development and quality assurance and finally with the same effort applied to the maintenance and operations of a running system. From the deployment of patches and extensions, to the cleanup and data hygiene of databases to the scaling of the system and hardware and software maintenance efforts.

As much time and effort is spent in the art of software development, often not much effort is spent during the operations phase of a software project.

A rule I’ve often observed in many many projects I’ve been part off is the linear degradation in the effort of maintaining all the scaffolding and activities which are not directly related to coding of a software project. Thus enabled entropy degrades the product and seeps into its operations lifecycle resulting in emotional and business drawbacks for both the organization and the stakeholders included in the project.

Should we accept this? Should we accept a system of development with diminishing software engineering efforts , should we accept the fallibility of people to care and work at top quartile engineering capacity?

I’ve seen that any significant software system, shop and development environment needs guardians in the form of an engineering educated project manager and a strong cadre of technical leaders which have the authority and will to enact and drive and force a strong commitment to all software engineering scaffolding practices which combat the intrinsic entropy found in all human endeavors, especially software development with its heavy intellectual burden, high abstract concepts and emotional stresses.

A common image of software developers is of introverted emotionally challenged people. Truth be told I’ve worked with quite a few of them in my time. Thus a surprising aspect of our live is the emotional toll software development takes from each of us which is directly linked to the quality of work we do and the state of the system.

Human emotions, feelings and states directly transfer to the work we do. The pain of working with  highly abstract concepts, what in truth software development is, is the primary source of all the bad decisions, all the accumulated technical depts., all unsolved problems and defects a system has.

In order to ensure a proper functioning of a system and a team special need needs to be put in and placed under constant guardianship in order to focus on emotional and mental health of the team working under such adverse, but normal conditions.

Code archaeology

 

 

There is no love, no tenderness, no emotion, no trust and no God in a piece of code. You can search, you can dig and you can wail at the injustice of it all but the only thing you are going to find are mistakes, abandoned paths, and lines upon lines of code written by people who are not here and all that they have felt, experienced during the creation of the code you are reading is forever lost, forgotten, not important.

But it is important. Recently I’ve done a lot of code archaeology, reading and analyzing solutions (some which are more then seven years old) discovering how such system are built and how are they organized with the goal of identifying pieces of code which can be transferred into the new ports of those system which are going to be built.

By reading that code I’ve found a story. A story of ideas not reached, of pressures and short cuts taken, of principles and practices which were a standard of the time that code come came from but now I know them only from my books and my own early work.

I was lucky and got to talk to the people who worked on those systems, and got to ask my questions. Why such a decision was made, why this and why not that. Is this used and why not. And I got a story of humans. People working and feeling their work end expressing them selves trough code.

The end result of each code archaeology activity was a new architecture document document the found architecture, all the technical decisions and patterns found within without judgment, written just like it is.

While doing it I found the following principles helpful:

• Do not judge
• Identify design pattern and name them
• Identity individual sub architectures and follow them separately
• Sketch

Do not judge is in my opinion the most important rule, principle and state of mind while doing code archaeology. The basic principle of the activity is to see and read what is built. The design patterns, architecture and coding style may well be something which is archaic, non modern and not pretty but it may not have been like that when it was built A web from application written in Visual Basic for .NET 1.0 is a child of its times and it is not reasonable to apply todays software development criteria to such an old system. Secondly I never met a developer who when faced for the first time with a system written and developed by someone else who didn’t start cursing and wailing about the poor quality and decisions made by the previous developers.

We developers are an opionanted bunch who would rather spend days and week arguing about what about the proper placing of business classes and their proper naming rather of accepting the current situation and getting some work done. This is actually good since is brings us to question everything and to advance as a industry as a whole

But when doing code archaeology, when reading and analyzing someone else's work this attitude will block us from seeing and understanding the software we are working with and finding it inner beauty.

The most important contribution to the modern practice of software development was the definition and introduction of Design patterns as a vocabulary and definition language which transcends all languages and software types. Thus when doing software archaeology it is most important to identify patterns of code use, name them accordingly and document their use. This practice makes the activity of analyzing and communicating the results of a software code archeology that much easier.

Any non trivial software project is actually composed of many individual sub architectures which together bring the solution to bear thus as an individual developer sees his work trough the integration of a group of classes thus a software architect sees a software system as an interaction between multiple sub architectures.

Not only it makes easy to document a complex software solution, it is the only easy way to analyze it. Modern software is complex. It has multiple moving parts which interact with each other trough time and space. A human mind can fathom only a limited set of the reality it is immersed into it at any given moment of time. Thus the only logical solution to comprehensively analyze any software system is to follow to exclusion of everything else each sub architecture boundary within is self separately from every other architecture.

Then, and only then, after each sub architecture was analyzed to its full extent. The next step is to analyze the interaction between various architecture, the same as we watch the interaction of several classes.

In the end software archeology is about understanding, documenting and the future. As archaeologist catalogue, analyze and document their findings for other to use, thus a software archeologists end product is document describing her finds. The best tool at the disposal of software archaeologist is the ability to visualize graphically old the processes, integrations and logical modules of all the key ideas being documented.

Living somewhere else

 

Living in a different country it’s a strange occurrence. Feeling of fear, insecurity and danger are more accented, more real, more tangible. It becomes easier when some of the basic living necessities are handle. Finding friend, having a place to stay, a place to eat and a place to drink bear.

Meeting someone new also helps.

On war and peace

I’ve heard of MakerBot and the cheap 3D printing movement a few years ago by listening to a FLOSS weekly episode by driving on my daily commute. For me it was far away, costly and not something to think about. But listening to a couple of Dot Net Rocks shows and watching an Elementary episode I came to realize 3D printing may well starting to affect our lives, and it in the next five years it will greatly influence how we live. And not always in positive ways.

The internet was always full of dangerous things, the anarchist cookbook was one of the early examples. People could download it and learn how to make bombs. Want to hack a pay phone, you can find it on the internet. Learn how to pick a lock or steal a car. Find it on the internet. Learn how to torture someone. Internet.

Learn how to make gunpowder. Well on Wiki how you have picture by picture instructions even my six year old can follow. And once again, the internet.

Of course the majority of the people don't do that. Most people are not interested in those stuff. For me I’ve learned how to make a poncho from fleece, how to fix a broken sink and how to roll a cigarette.

I’ve read this article on endgadget. And I was scared. Home made guns. Readily available to everybody, without supervision and control.

I picture kids going to school with weapons, with sharp plastic knives and maces. With crossbows and other stuff they’ve printed on their home printer. Metal detectors would not detect them, well they are made of plastic.

The world just got scarier.

I and the majority of my generation have survived a war. We have witnessed first hand what war and weapons do to people and their lives. Thus I’m really glad I leave in a country where the majority of the population is legally disarmed and the right to bear arms is not in place.

I believe that firearms and lethal weapons have no place in modern society. What would be the use for them? We are not being invaded, or have to defend our selves by murdering war bands or have to survive a zombie apocalypse.

There are dangerous people out there. deranged, sad people with issues. But they are far an few between and you and me know where those people can be found and any sane person knows what to do to avoid them.

When I first heard of mainstream availability of 3d printing I thought “Oh, wow, how cool! Now I can make toys for my boy and Warhammer 40k figurines for me”. Predator Annihilator here a come. A 2000 pt Ultramarine army right at my fingertips. Special terrain, bunkers and all the things we made from scrap before just waiting to be printed and be used.

If I was thinking of making a weapon with a 3d printing I wouldn’t create a gun. A crossbow and darts would actually be better for me or short sword, mace or morning star. And lets go LARPing.

But seriously I wouldn’t do any of those things. I would think I have a weapon, but to use a weapon is to kill a person. I do not need nor whish to kill a person. I don’t really relish going to prison and do not really think about a scenario where I would need to constantly armed.

The place I currently live is filled with rolling hills and valleys which once were diving lines between great empires. In ages past the area around me was filled with fortresses where armies clashed, and maundering parties were common. Byzantines versus the Lombards, the Franchs, the Romans, the French , the Turks , the Germans and the Italians, the Americans. I live in the most peaceful battlefield in the world.

I wish it to remain so. I wish people would not make plastic guns at home. I know people in general can not be trusted with other people lives and safety, that’s why we have so many checks and balances.

I drank coffee today and watched clouds roll bellow me, following the Mirna valley hugging the land and covering bright lights for houses scattered across hills and forests. An artist discussed her new exhibition and people from all over came to see my land. In peace.

On time

 

Time is everything to a software developer. It bounds every aspect of his life. The time to come to work, the time of work and the time after work. But more to the point the time it takes to complete work – concentration – rest cycle, the number of times such a cycle can fit in a work day. The time lost talking to each other, eating, going to the bathroom, phoning, compiling, publishing, reading, searching waiting.

Most do not notice such a thing as time spent per activity, nor the time ill used or lost between meaningful work. Some do, and thus try to leverage it, learn it and thus they try to learn about them selves and their relation to the dimension of time

I’m such a person and I’ve learned a fair share of things in my time as a developer. I’ve learned that time is a feeling and that not all time can be meaningfully spent. That you are bound to have two to three work-concentration-rest cycles a day and that is that. I’ve learned that you can leverage the time between to better your self, watch a presentation or read an article during compile times or listen to podcasts on your way to work.

Such activities seem trivial when taken by them selves but focus on that, those five minutes spent listening to a talk on infoq are five minutes that you are wiser and potentially know more then before, albeit it was five minutes worth and if those minutes pile up during a month or a year how much time did you spent bettering your self instead of reading comic books, facebooks, twitters or other useless information's?

But other than that time is crucial for the activity of software development. The mastering of time by individual contributors which need to give estimates on their work and technical leads who need to coordinate the continuous integration of different modules in order to utilize the full potential of the team and incur no loss in output and productivity to product managers who need to be responsible for the deadline, or at least know what will be realistically produced by when.

When doing the work of a technical lead in charge of leading a team of developers time synchronization is of the up most importance. Imagine for a second a team of four developer plus out valiant imaginary leader. They are building a distributed system which will constructed of two main parts.

The first part is a a server component exposing some kind of fanciful services over the network. That server component is built with the goal of serving many clients build by potentially different organizations in different technologies.

The second part is a client application built to use the services exposed by the server component.

Our valiant leader thinks for a seconds and break the team in two parts. Two developers will work on the server component and two developers will work the client component.

By sheer luck the specification for the server interface is completed and hopefully will not change during development (one can hope at least). The client team is crack A-team of mixed UI design and Software development specialist, who know all there is about human machine interaction, how to draw fancy stuff and all there is about design patterns and software development.

Since they must build the client without the server component to test on they’ve leveraged their design pattern knowledge (with the help of our valiant and heroic technical leader) and they we pushed the server interaction code into their lower most layer and mocked up the input and outputs. Thus the super uber client is build. But alas the server component hit a glitch, it seems more people were needed so while the client is fully built but not tested on a real server the server component is build and tested only up to 30%.

Jeez what now?

Described above is problem of resource allocation and timing. What could be done different? Well a lot of things, the client work could have been started later or the team should not be broken into two separate sections instead each developer or pair could have been given a cross section of client and server functionality thus the client and server parts would be build, tested and rolled at the same time.

In end time is the king that marks each cycle in the life of a software developer. Knowing its rules and limits, its ebbs and cycles it is crucial for a developer to be successful in her career.

On the life

 

As software engineers our life are subject to forces which are not entirely in our control. The chief are health, family and politics. I will start from the latter and go the former, discussing how a software engineer can mitigate some of their ill effects on her life and career.

Politics is the art of the achievable, of getting things done. I’ve listened to an audio book version of the ‘Prince"’ by my namesake Nicollo Machiavelli. While some may disagree with its context or rebuff it in its entirety on moral grounds alone, I generally find its context true and suitable to any endeavor where you have a hierarchy of people working in environment with limited resources, or any environment where you can find humans. The truth is that in the base of the human nature lies the instinct of self preservation, which is often achieved and perceived to originate from ones advancement above her station in respect to the domination and control her peers. When working in any organization there will always be personal goals, issues, treats originating from people working around you, above you and bellow you.

‘I just want to do my work, and be done”, some may say. And you are obviously right in that life philosophy, but given the fact that out there are people who will benefit by inflicting or just by you incurring a loss of some kind. Thus a wise person should strive to position him self in a prepared state, watchful about what is going on around him, about the goals and positions of the people in his circle. He should align him self with the people who have ambitions in alignment with his own, make useful and not a threat to people above his station and respected and not an obvious impediment to people bellow his station. A wise person would also be aware that each state is only temporary and thus not permanent and that person will always need to be on the lookout for the next situation and have always a backup plan when all things start to fall apart.

In short going to work nine to five and forgetting about work afterward is not a really good strategy any more.

The second ill which may befall a software engineer is family. I’m a family man, my family is my sanctuary, my church and holy ground. I short my family is my life. But often the work we do puts us across the needs and wants of our family members. Our spouses, our fathers and mothers, out girlfriends. This is the truth.

If you read any book about people, or software processes or any topic related the the culture of software engineering you are going to pass by references to “Death march “ projects, long hours and weekends spent working, broken projects and divorces which resulted from those activities. The pressures and the time, of the time, we spend working maniacally in order to reach our deadlines reflect negatively on the needs and goals of those around us. No amount of money or bonuses is going to mitigate that. Maybe you live on a farm and regardless of your work you must spend time working on the family estate (new a guy who had a problem like that) or your wife is fed up being all alone with the children while you do whatever you do. And then meeting your business commitments and your family commitments is going to get really though, and your morale and your performance is going to drop and your private life is going, well down under.

What can a savvy engineer do than? First, you can’t please everybody. You have a limited resource, your time, and two opposing forces. What I’ve found out is that you need to be open at the beginning to both parties about what you can do and what you can’t do. And then be prepared to take blows. Or marry a wife who will not mind you being around so much. Or find a nine to five job.

The third and most important is your health. Your mental and physical health is the most important assent you are having in your career as a software developer. In order to achieve the level of performance and excellence you are able and required to produce you must be mentally and physically fit. Thus you must always leave room for physical exercise and strive to conserve a positive outlook to every situation and prospect. I can’t remember how many time those two thins helped me, and I can remember oh so well how many time one or the other failed me.

Experience is never taught, only learned by doing. Thus read and watch what others have done and experienced. A good documentary I watched five years go showed a year of frenzied development of a team of engineers at Netscape developing the first version the Firefox open source browser. That is a good example of what kind of life we lead.

But all is not dark, and all is not done, for we are human and we make mistake but to be human is to be able to learn and change, to adapt and improve thus we will make our lives better trough knowledge, spiritual advancement and physical toil.

In the beginning there was the command line

 

In the begging there was the command line, and it was good. Then came the user interface, the mouse and expediency of ease of use. The word was “Don’t let me think” and thus the web was begotten.

And the cometh forth a beast of many heads, or many releases which swarmed into one version, and Google Chrome rose into the world and brought the command line back into the fold.

At first it was the ease of search, ease of address typing, moving and blurring the difference of an url and a query. Now, it was the turn of the information and the knowledge Google brought about your needs, your work, your life. And thus the time of search cometh to the world and the word was good and the masses were fed by the wine of information and food of quick results.

And the devil of engineering said “It shall not be enough, what was before shall come again, the circle of technology and innovation will turn again bringing what was before into the new and bright” and developers brought search to their applications and then it was not enough to have search but commands typed after the url came as orders of the divine and users could use the shell as the address bar and the web page as the console.

Thus in the end there was the command line. And is it was cometh again. Technology turns, what was old becomes anew, rediscovered, dreamed again.

I' am student of technology, servant of engineering and topologist of software. During my travels trough countless projects, books, presentations, frameworks, languages, databases spanning decades now I remember things which once were during the first books I’ve read at the end of the eighties and the nineties and the stuff I read and do now, and often I see old ideas and technologies rediscovered and re-implemented in new spaces.

I’ve read a book about the basics of computer and operating system architectures, how hardware and memory functions and how thing communicate and stood amazed how the same solutions and patterns are copied trough the architectures I’ve done and those I’ve dreamed.

I came to see this as truth about our loved profession. Everything which once was will come again, thus making the study of old systems, old patterns crucial for the future of our advancement. The critical thing which needs to be adopted as practice in our companies and universities and training grounds, and which is known to all old hands who survived several decades of shifting technologies and trend is that regardless that a technology is not used and more, or is redundant that there are core lessons and practices which will prove useful in the problem spaces of now and tomorrow. 

Silver bullet

When I'm truly happy at my work I find my self often thinking about the truth behind software development. I keep finding and formulating maxims to describe various states of my art. I like that.
I think myself as a philosopher of software engineering - looking for the truth in a methaphisical world of ideas and shapes.
I like working with people. But I'm not a social person. I prefer a quiet morning camomilla tea with a book in the office caffeteria then being outside and shooting breeze with my colleagues. Thus I find my self at a crossroad when I try to figure out my self in relation to my job.
My job has two dimensions.
One is a dimension of people and emotion, of tought and word, of smell and touch of ideas and egos. A world of chaos and entropy populated with - well, us humans or developers, IT people and managers all struggling like primordial cells to create something greater something in the face of constant enthropy.
The other is a world of clear shapes and vectors and models and pure crystalline tought. A world of boxes, layers hexagons and numerical formulas given shape inside our collective hallucinations, out matrix, of the systems we are trying to build, to document to give shape.
What am I? What do I actually do.
Nothing.
I just talk to people all day. I write documents all day. I draw diagrams all day. I write code all day. I dream stuff all day. I'm Ada Lovelace's son. A software engineer.
The truth is that I'm not really sure what I am. I think I'm searching for the truth. A unique way of developing software, the ultimate architecture - our silver bullet.
Brooks said that there are not silver bullets, there is not a perfect solution which will work for every project. I know that. I belive that. But still I carry withing me a germ which tells me to keep looking, to keep working on the perfect ultimate architecture.The perfect way of constructing software.
With people.
Software is all about people. In order to build the perfect architecture I need people. At least I need my self. And I'm not perfect. I go hungry.I get sloppy. I forget. I make mistake. I swallow the shame of dirty code and make out stories of better futures to hide out my mistakes from the dream of perfect ideas I constantly have.
Code is dirty. C-O-D-E is a four letter world. Code is not perfect, code is limited. You cannot represent the metaphysical with words and numbers. You can only try and fail. It is why so many of us quickly forget our past projects and greedily go to our next adventure. Next time will be better. But it never is.
I keep missing something in this mix of juices and ideas that we call software engineering. A deep idea, a connecting string. The ultimate truth.
A silver bullet. The all god spark which will give meaning too all there is and ever will be. From the first loop that counts time written ages ago to the modern fast growing, ultra delivering  systems of modern ages. Did developers of old knew something we didn't. Was software better in those days, more pure or was it always the same low level struggle from near emptiness to almost complacent skillfulness.
Code is like love, it is like work. A four letter word. It catches you and carries you unaware, you want it, you need it, you dream it and then you want nothing more then to run from it like the yesterdays girlfriend you are ashamed to wake up beside in the morning.
But like the proverbial secure bond of a known partner we come back to it and try to fail building perfect hallucinations of perfects forms with stone age tools and human minds.
What about God? Is he mad at us for dreaming new worlds, trying be like him. Like a Creator? Maybe we are just cursed, never destined to succeed. Always trying, always failing our sweet silver bullet always around the next corner. Oh, how I search for it so.