22 February 2015
Luckiliy with the the advent of oauth and other federation services nowadays I can signup form most services using my Google or Facebook account (the two I generally use mostly). One-click signup made my life easier.
A nice feature of using web services is the ability to reuse data, content and features from one service (e.g. Google Drive) within Another Service provider (Draw.io) and have changes to my work published to a group of people using a collaboration tool (Slack).
Most online service providers expose REST endpoints for almost every feature they provide. Some are notification features (reading statuses of artifacts) while other expose behaviors of the service (e.g. posting a facebook status or saving to my google drive folder or publishing a note on evernote).
Are we living now in the world of the programmable web where REST apis replace the old days of bash shell scripting in the unix command line for hackers, where a basic level of programming skills enables us to automate and integrate our online work life?
Tools like IFTTT or Zapier provide simple to use integration commandlets between various services which is a nice start for that entire space. But looking at their API's they are really simple. Comparing them to all the various possible REST apis published by almost everyone the amount integration and collaboration possibilities are enormous.
Leveraging various cloud based solutions its trivially simple to set up a continuously running background service in the cloud which executes various types of checks and integration tasks on a regular basis. I would recon that for lean starups and small organizations one of the first automation priorities they need to execute is to automate all their online services and collaboration tools.
New organizations are not just bound by one service provider but utilize a wide array of services offered on the internet, all of which can either be used for free or very cheap. Adding users and connecting all those services in one go is where most organizations will leverage the ROI of utilizing all those solutions.
24 January 2015
But there a silver lining. Bosnia is a micro representation of the European Union as a whole, a real life testing bed for European ideas and models. The EU has long and bloody history of conflicts,ethnic cleansing and old grudges popping up after five minutes of talking to a national of another EU country.
With so many languages and cultures living in the same place going about and handling business across the EU its not that easy. Especially the language issue. Lets face it , English is the lingua franca of the world and if you want to hop about the EU the only language you really need is English. It helps to know the local language but how can you expect a single person to be fluid in all the twenty something national languages of the union?
In Bosnia there are three main languages :
11 January 2015
The systems we brought into the world talked with the operating system using POSIX commands, with each other by sending messages over pipes, tcp channels and finally using the HTTP as the universal communication layer.
The world of modern application development is a world shaped by two opposing communication forces:
- our systems need to talk and have a meaningful relationship with humans
- our systems need to talk and have a meaningful relationship with each other.
- One pure logical model
- One or more external communication points (ports)
- Zero or more internal communication point (ports)
- One or more integration environments providing adapters for one or more external communication points and all internal communication points
- exposing one or more external points (ports) to different clients trough its adapters
- providing adapters for all internal communication points.
- Routing http requests
- Ability to work with Http Request and Responses on a high level
06 January 2015
Outsourcing is all about making the biggest return of investment (oh, well no big surprise there) by reducing as much the development time possible in order to satisfy all the key client requirements.
When approaching any outsourcing project you have the following guarantees:
- You are never going to know everything you need to build the project
- Some of the technology involved will be unfamiliar to you
- You are not going to have enough time.
- A self measuring and self correcting software development process
- Strong requirement gathering and managing skills
- Will to learn outside of regular working hours
- Role power
- Technical power
- Relationship power
- Lots of overtime
- Lots of technical dept
- Lower quality of the delivered solution
- Lower maintainability of the code
01 January 2015
Microservices architecture models - How to document and think about microservices based architecture models
For those who don't yet know Adrian Cockcroft is the chief cloud architect for Netflix, a company that revolutionized cloud architecture and in a cause reaction way started up the DevOps movement. Yep, one thing they don't tell you is that when you transition to an architecture based on visualization, stateliness, and computerization of business features you drive up your costs related to the management and development of tools related to your day to day operations. It really puts Dev into Ops. Gone are the days where you could just put and app in production and forget about it on a day to day basis. Nowadays you need twenty something tools just to run and manage everything.
Well, heck, that's not what I wanted to write about today. As you may have seen I've upped up my software engineering writing and profile these days. I've writing about other stuff lately (mainly on this blog Fighting fitness , with my fiancee, about my wonderful journey into self defense and losing weight) .
The first of January started with a well made plan to work on an project I started a couple of weeks ago. Yeah, I didn't. In exchange I've spent the morning trying to find a formal definition of the architecture model Cockcroft presented in his talk at Dockercon.
I couldn't find it, but instead I found a bunch of articles and resources about the current state of micro-services architectures in the wild.
I've been involved in a government project for the last year or so, changing and modernizing a national VAT system IT infrastructure. As part of the this project we build a service oriented (micro-service based) infrastructure with the goal to reduce the load on the main VAT system and ease the development of supportive business functions (e.g. bounded context) like a separate VAT Portal for Taxpayers or a system for VAT refunds for foreign companies. And we all did that with a fairly conservative .NET stack (its the tax authority, we live conservatively).
During his presentation Cockcroft has shown several architectural graphs of the current Netflix architecture. The looked like nineteen seventy fractals from a weed high math graduate from Berkley, all accompanied by a comment that it is impossible to show an architectural model for Micro-service based architectures.
Micro-services based architecture can generally be tough as :
- Application topology (fractals baby, fractals)
- Data processing and transformation system (Event driven, space and time unbounded ETL)
- Application architecture
- Tooling (basic development and deployment tools - think unit testing, packaging, integration)
- Configuration (with lots of servers, databases, services we need lots and lots of configurations)
- Discovery (with things popping up and dieing all around us in a really chaotic fashion we need to figure out how to discover things)
- Routing (back to the chaos of microservices, we really don't want to hardcoded every url and resource - think load balancing)
- Observability (we need to monitor and log what the hack is going with our process , logs and events and process tracing).
- Message queue (for eventing)
- Web API Gateway - for routing micro service calls (and serve as a circuit breaker in case of service failure).
- Raise "I'm here events" or "I'm dieing Jim" events trought the event system which will be read by the Web API gateway which will in turn manage the service registry (who is where)
- The web api gateway will queue service calls in case of failures (for important operations) so when our services fail we have a log of all calls made to them and we can execute them as soon as the failing service is up again (e.g. circuit breaker)
- Services need to handle data stored in other services (e.g. have their own local copies) so they need to publish their internal events and react to events published by other microservices.
- Web UI (Web aps instead of Controllers)
- Domain (Microservices instead of Business logic layer)
- Web api Gatweay, Message Queues and routers as the glue that holds everything together
- Entites and other domain goodies
- A way to publish services and connect to other services (we really do not need twenty way to access or read data from an internal web service)
- A way to publish and to react to events
- A way to interact with the Web API gateway
- A way to access the database (or database groups).
22 December 2014
As a software engineer I've worked on both brown field and green field projects. As a young engineer I thought that green field projects were the best. The freedom to experiment and do what I wanted to learn and use the technologies I needed to learn instead of the mess somebody left for me to work on.
And then I came upon a legacy brown field project. Large in scale and large in years with a rich and complex history behind it. With problems, with issues and design decisions that stuck and made life miserable for developers for years to come.
And I learned.
Oh, boy how I learned.
I would advise any young engineer to work maintenance on any large system for a year or so. The possibility for growth and learning are endless. There is no bigger learning incentive then adversity put upon your not so strong back.
On December 5th I've held a talk for the HTML5 User group in Banja Luka where I worked for a year or so. It was the second such talk. It seems they liked the first so they invited me for one more.
As a theme for the talk I played with the idea of extending the concept of brown field software development to organizations. How to develop and integrate new software into existing organizations, organizations which are not your own which at times may be adverse to you , have their own quirks and rules.
Not to bother any one, the gist of the matter is that there are three key elements, ideas, issues, topics, and agendas one must cater to when coming to work as an outside entity into an existing organization:
01 January 2014
The eternal truth to software development is its inability to produce any meaningful systems without a coordinated effort from a diverse team of individuals, lots of disparate activities ranging from design to development and quality assurance and finally with the same effort applied to the maintenance and operations of a running system. From the deployment of patches and extensions, to the cleanup and data hygiene of databases to the scaling of the system and hardware and software maintenance efforts.
As much time and effort is spent in the art of software development, often not much effort is spent during the operations phase of a software project.
A rule I’ve often observed in many many projects I’ve been part off is the linear degradation in the effort of maintaining all the scaffolding and activities which are not directly related to coding of a software project. Thus enabled entropy degrades the product and seeps into its operations lifecycle resulting in emotional and business drawbacks for both the organization and the stakeholders included in the project.
Should we accept this? Should we accept a system of development with diminishing software engineering efforts , should we accept the fallibility of people to care and work at top quartile engineering capacity?
I’ve seen that any significant software system, shop and development environment needs guardians in the form of an engineering educated project manager and a strong cadre of technical leaders which have the authority and will to enact and drive and force a strong commitment to all software engineering scaffolding practices which combat the intrinsic entropy found in all human endeavors, especially software development with its heavy intellectual burden, high abstract concepts and emotional stresses.
A common image of software developers is of introverted emotionally challenged people. Truth be told I’ve worked with quite a few of them in my time. Thus a surprising aspect of our live is the emotional toll software development takes from each of us which is directly linked to the quality of work we do and the state of the system.
Human emotions, feelings and states directly transfer to the work we do. The pain of working with highly abstract concepts, what in truth software development is, is the primary source of all the bad decisions, all the accumulated technical depts., all unsolved problems and defects a system has.
In order to ensure a proper functioning of a system and a team special need needs to be put in and placed under constant guardianship in order to focus on emotional and mental health of the team working under such adverse, but normal conditions.