Sunday, June 30, 2019

Challenges Facing IoT Systems in 2019



Challenges Facing IoT Systems in 2019

There is no doubt that we have made significant progress in many key components of IoT systems, in recent years. These developments have in fact made IoT possible. Computer hardware has come down in cost, size and price. For many the Raspberry Pi and other similar devices has facilitated the ability to experiment and innovate, this coupled with the huge open-source community that has provided operating systems and tools that unlock these devices.  

Mobile and other communication networks have improved driven by the demand from initially mobile phone users. This has also driven the development of mobile devices, which has driven improvements in batteries, screens, sensors and the entire software eco-system that runs on these devices. 

With all this development, there are still a number of significant challenges. In order to roll out significant quantities of IoT devices, there are opportunities to improve deployment solutions. we live in a world of orchestrated containers, whilst this has created significant progress in terms of continuous deployment these technologies are network intensive and will require further refinement to work in a performant way in an IoT context. Engineers in IoT face similar challenges to those who tackled the large data scale issues 10 years ago, however with more constraints. 

The logistics of deploying large volumes of sensors is prohibitive and hence dynamic software updates are almost mandatory.  

With the vast number of devices deployed, dynamic updates and the importance of the systems that become reliant on this data, security becomes a big concern. We need mechanisms that ensure data validity and authenticity in a way that is effective and performant.

Bandwidth is improving, but this is being consumed by the volume and the number of applications for IoT data. Hence bandwidth remains and will possibly always remain a constraint. 

With the proliferation of devices, we have challenges around processing this data. Technology is emerging that can capture and process huge volumes of data, however, the technology and tooling around these technologies requires further development. 

Hanging off all this data gathering, we are developing data models and machine learning models, there are significant challenges deploying these models. In many cases, we have sensor data, which need to be processed, this along with more static data, such as location, geographical, weather or other data.

Along with all of the above, we need systems to monitor the entire network. In many cases we cannot consider the IoT devices as "Edge" devices, rather they are becoming "part" of the network, and as such need a sufficient level of monitoring to ensure that the overall system remains reliable and stable.

Whilst IoT has certainly opened up the realms of possibility, it brings with it new challenges that we are yet to completely solve. We might look to how we solved similar problems in the past and apply some of that thinking to the new context, or we may have to look for new innovative approaches.

Monday, November 19, 2018

Developing on the Move


I am constantly amazed at the rate of progress in Software Development. Years ago, I recall writing tedious scripts using PERL, TCL and EXPECT to log into Switches and issue commands to provision them. Today we have Docker, Ansible, Kubernetes and a host of other tools, we have advanced compilers, transpilers, linters, formatters and various other tools. We can provision infrastructure on the cloud in seconds and then build complex systems quickly and efficiently. Obviously this means that the systems themselves can become ever more complex.

Computing power is such that AI now has practical applications, natural language processing, computer vision and other applications.

It occurred to me on my lengthy, rainy bus journey home this evening that the next step in our development evolution might be the ability to develop software and systems on the fly. I would love to be able to develop systems whilst I walked in the park or sat observing a lovely view. Would it be possible to talk to an interpreter and then run some process on the output to generate the code and the systems I am developing.

Presumably, we would need to define a language of common constructs that is extensible. Something like :


  • Project create
  • File create index javascript
  • Function create test
  • AWS Provision EC2

Each developer could then using some "modules" that where kept in a open-source repository run some parser on these instructions and produce some output, which would include code, in a language of their choice and DevOps in the technology of their choice. The system would need a language definition that could easily be extended 

Interested to hear the thoughts of others on this ...

Thursday, June 28, 2018

Caution: The DevOps explosion

Everything in moderation - Let's not overdo the DevOps !

Recently, working as a startup CTO and adviser, I have seen what I call a "DevOps" craze. Developers, especially younger ones seem totally swept up in the DevOps revolution. Continuous deployment, auto-scaling groups, hot back-ups, containers etc etc.

Many times these implementations, whilst impressive, are unnecessary for the stage of the company or the product development. Many times, they are difficult to document, relatively expensive, difficult to maintain and can be hard to debug. In many circumstances, it would be far easier to have some old-fashioned deployment or back-up scripts until the company or product has some traction in the market and there are the resources available to properly administer the complexity of the DevOps setup.  This is somewhat of a paradox, initially it would seem that by automating many of the mundane operational tasks that the development team face, you would be saving not only time and resource but also money. However, this is not always the case. Like anything you should implement exactly what is needed and not more. Many service providers make some of these DevOps tasks almost trivial to implement (individually) that the temptation exists to over-engineer the DevOps. This should be avoided.

The main reason to avoid over-engineering of DevOps is not the potential to loose control of cost, it is not the risk of spending much needed development time on DevOps tasks. The main risk is creating an overall system that is reliant on a number of DevOps tool providers, which when used in conjunction create systems that are unnecessarily complex to understand and maintain. For example you push your code to source control, which integrates with another 3rd party, this runs some unit tests and pushes to production servers, these servers have a number of containers that are "networked" together and there servers themselves have various firewall rules and scaling rules etc etc. One can see that this can get very complex, very quickly. Yes, it's wonderful to have these tools and in principle it is wonderful conceptually. But is it really needed ? How often do you really need to deploy to production ? How difficult is the deployment ? How many people take the time to properly document the DevOps configuration and setup ? Is DevOps becoming a risk to your business ?

If you are not carefully managing this, DevOps can become more of a risk than a asset to the business.

Sometimes, less is more. Everything in moderation - especially when it comes to software engineering. 

Saturday, June 2, 2018

Hire for Engineers not for Frameworks

I have been faced with hiring great engineering teams on a number of occasions, I am constantly amazed at home many hiring managers and recruiters seem to focus on particular technologies and experience in those technologies.

My experience has taught me to hire engineers and I have always viewed the technology as a secondary consideration. I always focuses on a grasp of engineering fundamentals and not on information about some current or past technology. In our current world these technologies are out of date by the time the ink on the contract is dry.

Recently, I was approached by a leading company for a very senior role. The in-house recruiter laid on his wonderful resume with world leading software companies before affording me the chance to explain my career-to-date. He then proceeded to ask me a number of questions on specific Java libraries and how they were used. This is all information that is readily available on reference sites and in API documentation and does not  highlight any engineering ability whatsoever.

Unfortunately, this practice is common place. Hiring tests are geared to determining how good someone is with a particular language or technology and not how good they are at thinking or learning a new framework or technology.

I have never hired teams this way. The best engineering or software teams are made up of good solid engineers, who can solve problems and can adapt to a very fast paced and moving technological landscape. 

Wednesday, May 16, 2018

The issues with GraphQL

I recently wrote a little about some of the issues with a technology like GraphQL. Whenever a new technology emerges that claims to solve age old problems, one has to look closely at the assertions made.

In the case of GraphQL the following is listed on their website:


  • Ask for what you need get exactly that
  • Get many resources in a single request
  • Describe what is possible with a type system
  • Move faster with powerful developer tools
  • Evolve you API without versions
  • Bring your own data and code
The How to GraphQL describes GraphQL as "the better REST"

  • No more Over and Under fetching
  • Rapid product iterations on the frontend
  • Insightful Analytics on the Backend
  • Benefits of a schema and type System
Whilst many of these assertions may be true, the reality is that they also can cause significant problems. Like anything in engineering, there are always trade-offs.

  1. Asking for what you need : The backend or API has to be able to generate the queries that will give back the data as requested. As your project grows, if not carefully managed, the queries get bigger and more convoluted - eventually slowing performance and becoming unmanageable.
  2. Getting many resources in a single request: Produces similar issues to the above, furthermore if not managed, can yield a system with many points of failure. If for example one of the sub-queries is not returning valid results or performing - the entire system can suffer (unlike REST).
  3. Evolve your API without versions: Whilst this would seem like a great idea, the reality is that in a complex graphQL implementation, if something goes wrong on the backend - it is much more difficult to debug and has the potential to significantly impact the system. You have to ensure that any changes to the API are backward compatible and don't break the system. You might once again revert to having versions.
  4. Bring your own data and  code: This alludes to the fact that the caller is oblivious to the backend data-source. No different to REST.
In the How To Blog Post mentioned above:

  1. No more Over and Underfetching: This is simply not true, front-end developers will always take the path of least resistance - often not considering performance. In a REST world they may have had to make multiple queries to populate a web page (for example). Since you are now providing a mechanism where this can be done all at once, it will happen. Over-fetching is a big problem in GraphQL implementations !
  2. Rapid Iterations: Yes, this is true - but the quality of the overall system is impacted.
  3. Insightful analytics: I found it much harder to measure the API performance in a GrpahQL world than in a REST world. Afterall, in a REST world one can simple analyse each route and easily find bottlenecks. This is not the case when there is single end-point.

In summary, whilst GraphQL is another tool in an engineering toolbox, it is not a miracle solution. If not managed carefully, it can become a problem in complex projects for the reasons mentioned above. Claims that are made by the creators of new technology must always be evaluated and not taken an face value.

Tuesday, November 15, 2016

The Myth of Machine Learning

Recently there has been a surge in the number of companies claiming machine learning capabilities, from startups to large organisations. The media is full of "machine learning" claims, investors seem to be dropping large amounts into startups claiming to have "machine learning" or "artificial intelligence" capabilities. From Logo design companies to delivery companies, they all claim to have implemented "machine learning". I recently saw a press release for a US based app development company that had raised a significant 7 figure sum, claiming they had developed "Human assisted machine learning" ! One has to ask, what is that ? Any machine "learning" has to be assisted by humans anyway, who would configure the algorithms, applications and hardware, not to mention the training ?

Neural Networks and AI genetic algorithms have been around for a long time. So why now ? The answer is data (and to some extent computing power).  In order to even attempt to do anything smart with "learning algorithms" one must have large sets of data. Since "Analytics" we have that data, sensors and devices connected to the Internet and mini-computers in our pockets, always connected. So gathering data is not a problem. The problem is what to do with the data.

Fundamentally (despite outlandish claims by the media) , Neural Networks can be "trained" to classify sets of input data into categories.


The diagram above shows a plane in 3D space that separates two sets of data (classification). If we project that plane into 2D space we have a non-linear equation.




If there is a means of determining if the classification was successful or not the classification can "learn" as the input evolves. Most current claims of "machine learning" are really simple rules that are able to cluster input data, there is nothing much too these claims. These are more like the old "expert systems" which have a pre-programmed set of rules that help the computation determine some output result. You can imagine this as a large set of IF THEN statements. It is a myth that these systems can learn or that there is any form of "intelligence" in these systems. Somehow we have jumped from these very limited capabilities to machines running the Earth. I guess the one thing we can learn from the media of late (BREXIT and Trump) is that you can't believe what you read !

Whilst I am positive that there are some very clever people out there working on all sorts of clever algorithms, I feel that propelled by media hype there is a big myth surrounding Machine Learning. In some ways it is in the interests of academics, business leaders, marketeers ect. to promote this myth as this fuels their funding.



Thursday, June 16, 2016

The CTO journey: 5 Things I have learnt

I remember hearing a quote from Herb Cohen, the respected US negotiator. It went something like "I never get called when everything is going well, I dunno who get those calls, but it ain't me !". This is true of many startup CTO's. Usually there is some failed outsourced relationship, or a technical co-founder who takes the product to a point and then gets stuck.

I have received many calls over the years and the problems are usually similar. This was indeed the case at my current position. Here are some things I have learnt ....


  1. Focus on the core product: There will be a lot of noise around the product, features that are not essential, wish lists, dreams. Stick to the basics ! Unless you get the fundamentals sorted out there is no future for the product. This requires single-mindedness, and possible a few "debates", the founders will want to get as many features in the product as possible. Don't give in ! The main role initially is knowing what NOT to do ...
  2. Mostly use tried and tested technology: When you are launching an MVP, or you have to deliver a core product quickly. Use technology that is well established and well documented. The experimental innovative stuff can come later. There is still plenty of room for innovation withing the confines of established technology. 
  3. Solve first - automate later: Often there are great ideas/features that will automate some process or some workflow. Often these can be done manually or can be done for example by writing some simple scripts that aid the process. Get these "manual" or "semi-automated" ways of doing things done first before jumping in and automating. Initially, there is often no need, no scale and within a changing environment it does not make sense to automate processes that are constantly changing or evolving. This can come later. 
  4. Protect your developers: whether you have an outsourced relationship or an in-house development team. Developers, designers etc get fatigued by uncoordinated feedback. Make sure that feedback (bug reports etc.) are handled in a coordinated fashion. The quickest way to destroy job satisfaction is by allowing everyone in the team to point out deficiencies. Also if you are outsourcing development, you want to keep the relationship sound. Protect this relationship. 
  5. Manage expectations: It is always good to over-deliver and under promise. Be realistic and communicate this at all times. So many times problems are caused by engineers saying what they think they should, as opposed to what they know they should.

Writing this today, I feel as if yet another journey has been undertaken and I start the next phase with our team at Knowledgemotion. We have secured investment from Ingram and signed a major deal with Pearson. They next phase will be taking the core product further, and hopefully beyond and expectation !