Thursday, June 28, 2018

Caution: The DevOps explosion

Everything in moderation - Let's not overdo the DevOps !

Recently, working as a startup CTO and adviser, I have seen what I call a "DevOps" craze. Developers, especially younger ones seem totally swept up in the DevOps revolution. Continuous deployment, auto-scaling groups, hot back-ups, containers etc etc.

Many times these implementations, whilst impressive, are unnecessary for the stage of the company or the product development. Many times, they are difficult to document, relatively expensive, difficult to maintain and can be hard to debug. In many circumstances, it would be far easier to have some old-fashioned deployment or back-up scripts until the company or product has some traction in the market and there are the resources available to properly administer the complexity of the DevOps setup.  This is somewhat of a paradox, initially it would seem that by automating many of the mundane operational tasks that the development team face, you would be saving not only time and resource but also money. However, this is not always the case. Like anything you should implement exactly what is needed and not more. Many service providers make some of these DevOps tasks almost trivial to implement (individually) that the temptation exists to over-engineer the DevOps. This should be avoided.

The main reason to avoid over-engineering of DevOps is not the potential to loose control of cost, it is not the risk of spending much needed development time on DevOps tasks. The main risk is creating an overall system that is reliant on a number of DevOps tool providers, which when used in conjunction create systems that are unnecessarily complex to understand and maintain. For example you push your code to source control, which integrates with another 3rd party, this runs some unit tests and pushes to production servers, these servers have a number of containers that are "networked" together and there servers themselves have various firewall rules and scaling rules etc etc. One can see that this can get very complex, very quickly. Yes, it's wonderful to have these tools and in principle it is wonderful conceptually. But is it really needed ? How often do you really need to deploy to production ? How difficult is the deployment ? How many people take the time to properly document the DevOps configuration and setup ? Is DevOps becoming a risk to your business ?

If you are not carefully managing this, DevOps can become more of a risk than a asset to the business.

Sometimes, less is more. Everything in moderation - especially when it comes to software engineering. 

Saturday, June 2, 2018

Hire for Engineers not for Frameworks

I have been faced with hiring great engineering teams on a number of occasions, I am constantly amazed at home many hiring managers and recruiters seem to focus on particular technologies and experience in those technologies.

My experience has taught me to hire engineers and I have always viewed the technology as a secondary consideration. I always focuses on a grasp of engineering fundamentals and not on information about some current or past technology. In our current world these technologies are out of date by the time the ink on the contract is dry.

Recently, I was approached by a leading company for a very senior role. The in-house recruiter laid on his wonderful resume with world leading software companies before affording me the chance to explain my career-to-date. He then proceeded to ask me a number of questions on specific Java libraries and how they were used. This is all information that is readily available on reference sites and in API documentation and does not  highlight any engineering ability whatsoever.

Unfortunately, this practice is common place. Hiring tests are geared to determining how good someone is with a particular language or technology and not how good they are at thinking or learning a new framework or technology.

I have never hired teams this way. The best engineering or software teams are made up of good solid engineers, who can solve problems and can adapt to a very fast paced and moving technological landscape. 

Wednesday, May 16, 2018

The issues with GraphQL

I recently wrote a little about some of the issues with a technology like GraphQL. Whenever a new technology emerges that claims to solve age old problems, one has to look closely at the assertions made.

In the case of GraphQL the following is listed on their website:


  • Ask for what you need get exactly that
  • Get many resources in a single request
  • Describe what is possible with a type system
  • Move faster with powerful developer tools
  • Evolve you API without versions
  • Bring your own data and code
The How to GraphQL describes GraphQL as "the better REST"

  • No more Over and Under fetching
  • Rapid product iterations on the frontend
  • Insightful Analytics on the Backend
  • Benefits of a schema and type System
Whilst many of these assertions may be true, the reality is that they also can cause significant problems. Like anything in engineering, there are always trade-offs.

  1. Asking for what you need : The backend or API has to be able to generate the queries that will give back the data as requested. As your project grows, if not carefully managed, the queries get bigger and more convoluted - eventually slowing performance and becoming unmanageable.
  2. Getting many resources in a single request: Produces similar issues to the above, furthermore if not managed, can yield a system with many points of failure. If for example one of the sub-queries is not returning valid results or performing - the entire system can suffer (unlike REST).
  3. Evolve your API without versions: Whilst this would seem like a great idea, the reality is that in a complex graphQL implementation, if something goes wrong on the backend - it is much more difficult to debug and has the potential to significantly impact the system. You have to ensure that any changes to the API are backward compatible and don't break the system. You might once again revert to having versions.
  4. Bring your own data and  code: This alludes to the fact that the caller is oblivious to the backend data-source. No different to REST.
In the How To Blog Post mentioned above:

  1. No more Over and Underfetching: This is simply not true, front-end developers will always take the path of least resistance - often not considering performance. In a REST world they may have had to make multiple queries to populate a web page (for example). Since you are now providing a mechanism where this can be done all at once, it will happen. Over-fetching is a big problem in GraphQL implementations !
  2. Rapid Iterations: Yes, this is true - but the quality of the overall system is impacted.
  3. Insightful analytics: I found it much harder to measure the API performance in a GrpahQL world than in a REST world. Afterall, in a REST world one can simple analyse each route and easily find bottlenecks. This is not the case when there is single end-point.

In summary, whilst GraphQL is another tool in an engineering toolbox, it is not a miracle solution. If not managed carefully, it can become a problem in complex projects for the reasons mentioned above. Claims that are made by the creators of new technology must always be evaluated and not taken an face value.

Tuesday, November 15, 2016

The Myth of Machine Learning

Recently there has been a surge in the number of companies claiming machine learning capabilities, from startups to large organisations. The media is full of "machine learning" claims, investors seem to be dropping large amounts into startups claiming to have "machine learning" or "artificial intelligence" capabilities. From Logo design companies to delivery companies, they all claim to have implemented "machine learning". I recently saw a press release for a US based app development company that had raised a significant 7 figure sum, claiming they had developed "Human assisted machine learning" ! One has to ask, what is that ? Any machine "learning" has to be assisted by humans anyway, who would configure the algorithms, applications and hardware, not to mention the training ?

Neural Networks and AI genetic algorithms have been around for a long time. So why now ? The answer is data (and to some extent computing power).  In order to even attempt to do anything smart with "learning algorithms" one must have large sets of data. Since "Analytics" we have that data, sensors and devices connected to the Internet and mini-computers in our pockets, always connected. So gathering data is not a problem. The problem is what to do with the data.

Fundamentally (despite outlandish claims by the media) , Neural Networks can be "trained" to classify sets of input data into categories.


The diagram above shows a plane in 3D space that separates two sets of data (classification). If we project that plane into 2D space we have a non-linear equation.




If there is a means of determining if the classification was successful or not the classification can "learn" as the input evolves. Most current claims of "machine learning" are really simple rules that are able to cluster input data, there is nothing much too these claims. These are more like the old "expert systems" which have a pre-programmed set of rules that help the computation determine some output result. You can imagine this as a large set of IF THEN statements. It is a myth that these systems can learn or that there is any form of "intelligence" in these systems. Somehow we have jumped from these very limited capabilities to machines running the Earth. I guess the one thing we can learn from the media of late (BREXIT and Trump) is that you can't believe what you read !

Whilst I am positive that there are some very clever people out there working on all sorts of clever algorithms, I feel that propelled by media hype there is a big myth surrounding Machine Learning. In some ways it is in the interests of academics, business leaders, marketeers ect. to promote this myth as this fuels their funding.



Thursday, June 16, 2016

The CTO journey: 5 Things I have learnt

I remember hearing a quote from Herb Cohen, the respected US negotiator. It went something like "I never get called when everything is going well, I dunno who get those calls, but it ain't me !". This is true of many startup CTO's. Usually there is some failed outsourced relationship, or a technical co-founder who takes the product to a point and then gets stuck.

I have received many calls over the years and the problems are usually similar. This was indeed the case at my current position. Here are some things I have learnt ....


  1. Focus on the core product: There will be a lot of noise around the product, features that are not essential, wish lists, dreams. Stick to the basics ! Unless you get the fundamentals sorted out there is no future for the product. This requires single-mindedness, and possible a few "debates", the founders will want to get as many features in the product as possible. Don't give in ! The main role initially is knowing what NOT to do ...
  2. Mostly use tried and tested technology: When you are launching an MVP, or you have to deliver a core product quickly. Use technology that is well established and well documented. The experimental innovative stuff can come later. There is still plenty of room for innovation withing the confines of established technology. 
  3. Solve first - automate later: Often there are great ideas/features that will automate some process or some workflow. Often these can be done manually or can be done for example by writing some simple scripts that aid the process. Get these "manual" or "semi-automated" ways of doing things done first before jumping in and automating. Initially, there is often no need, no scale and within a changing environment it does not make sense to automate processes that are constantly changing or evolving. This can come later. 
  4. Protect your developers: whether you have an outsourced relationship or an in-house development team. Developers, designers etc get fatigued by uncoordinated feedback. Make sure that feedback (bug reports etc.) are handled in a coordinated fashion. The quickest way to destroy job satisfaction is by allowing everyone in the team to point out deficiencies. Also if you are outsourcing development, you want to keep the relationship sound. Protect this relationship. 
  5. Manage expectations: It is always good to over-deliver and under promise. Be realistic and communicate this at all times. So many times problems are caused by engineers saying what they think they should, as opposed to what they know they should.

Writing this today, I feel as if yet another journey has been undertaken and I start the next phase with our team at Knowledgemotion. We have secured investment from Ingram and signed a major deal with Pearson. They next phase will be taking the core product further, and hopefully beyond and expectation !

Sunday, May 15, 2016

Shold Cloud Computing Costs be Regulated ?

I am not one for bureaucracy, in fact the contrary. I cringe at unnecessary bureaucracy and regulation. The face is that we are becoming a more highly regulated society, that being stated, I do feel there are some areas where we need regulation to protect the consumer. One such area is the regulation of costs by commodity suppliers, telecommunications, energy, fuel etc.

As CTO of a emerging start-up I am responsible for ensuring that our systems run on the latest and most secure technology.  Cloud services provide that scalable infrastructure and so naturally I use them extensively. However, let me tell you a story of one warm Friday night last July.

I was just about to go on paternity leave and settling down in my lovely new bath for a Friday night soak, when the phone rang. A chap was phoning from Amazon Web Services (AWS). He politely enquired if I was aware that our company had just incurred over $20 000 dollars of charges transferring some data from Glacier to S3 storage. My heart skipped a few beats, at that stage in our companies life the $20K would almost certainly bankrupt us and leave me without a job on the eve of the birth of my son ! Needless to say, I expressed my horrer at this news and began the long-winded process of composing various emails to Amazon to recoup the money.

Earlier in the day, I had read the AWS pricing guides relating to Glacier. I found them to be rather complex but I ploughed through. I have some ability in mathematics and two engineering degrees, I thought might assist me in understanding how to calculate the cost of transferring some data. I took out a pen and paper and after a long-winded calculation came up with a figure of around $300. I double checked this and when I was certain instructed my team to make the transfer. Obviously I got the calculation slightly wrong, by a factor of about 100 !

This is not an isolated case, I have heard of many very qualified engineers tripping up at computing cloud computing costs. This is because the pricing always involves a number of very elusive quantities, data transfer per unit of time. CPU minutes used etc etc. These quantities are very hard to calculate a-priori and have never previously formed part of and computational analysis.

I think it is time that our regulators take a look at these "Cloud services" and set some guideline on fair and understandable pricing. Afterall Cloud computing services are just another commodity supply that is crutial to the way we live and do business. We cannot be (a) held to ransom and (b) confused to the point where we are unable to accurately calculate the future costs.

The story ends well, AWS refunded the money. I have been slowly transferring my data ever since.

Thursday, July 24, 2014

Implementing Product Development Methodologies across Cultures and Organizations

Implementing Product Development Methodologies across Cultures and Organizations

After experience at trying to bring various methodologies and ways of approaching product development to different types of organizations in different countries, and often not succeeding as well as I would like, I thought I would attempt at finding some explanations.

I currently think of 3 dimensions that are the most important in terms of how they impact a generic methodology or approach. This is not an exhaustive list, however, after much thought these are what I feel are the key contributors to success or failure of putting an effective methodology or approach in place within an organization. 

Communication

The way that people communicate within the organization or team significantly affects the process of product or service development. Different cultures communicate in different ways, some more "open" others more "closed". Some "hierarchical" others less so.

Language can also be a problem, often nuisances are lost in translation. The same holds true when communication is between differing teams within an organization, i.e between the technical staff and the marketing department.

Key dimensions of communication:

  • Open or closed communication
  • Inter departmental or organizational "translation" issues
  • Hierarchical communication path
  • Conflictual or Non-conflictual 

Planning

Planning is related to communication, but the way in which planning is conducted and the hierarchy of stakeholders, how they get involved and who has the final say, differ. Also the way in which different cultures follow a plan differ, some might use it as a "loose" guide and others might use it almost in a "biblical" fashion.

Some organizations skip the planning stage or do not believe that it is needed or productive. Others won't do anything without a comprehensive plan, they may be paralysed by "not knowing".

Key dimensions of planning:

  • Valued or Avoided ?
  • Loose guide or biblical ?
  • Rapid or slow ?

Decision Making

Who makes the decisions is very important, in some cultures the leaders has the final say and is not argued with, in others he is guided by staff. In some cultures even if an idea comes from a team and is well thought out, any idea that comes from senior staff is immediately seen to be superior (observations in Brazil). Other cultures need consensus (observations in Japan) and things cannot progress until the entire team is convinced of the idea.

Key dimensions of decision making:

  • Leader driven or consensus
  • Logical or political
  • Rapid or Slow

Summary