Insights & Suggestions For Effective Software development
I decided to note a few observations on software development based on my experiences with a number of clients over a long career, this is work in progress and would appreciate any further input or comments.
Topics
- Data science needs systematic approach
- Without Specification you are in trouble
- Interview for excellence
- Agile does not mean "do nothing"
- Devops first is not always productive
- Too many resources is as bad, or worse than too little
- Design before Develop, before Deploy
1. Data Science needs a `Systematic Approach`
We live in a `Data Science` age, it's a buzz word. We have always had data science in some form or another, it's just that the availability, techniques, compute power and volume has exploded over the last few years. Principles are not new, they are standard engineering practises, however, in the new generation of `Data Science` these standard practises are often overlooked.
Here are some general rules-of-thumb I have noted down :
- It makes no sense to use machine learning to correct bad data that can be corrected by improving the data sources themselves. Machine learning in general should be used to clasify data to gain insights.
- Machine learning or other `learning` techniques are not guaranteed to converge, you should use these techniques with caution, certainly should not rely on them as a primary source of truth.
- Models work under certain conditions, you have to take the superset of possibilities into account as well.
- Developing an algorithm that works on a sample clean data set, is only the beginning of the problem. Running it in production with real data (maybe real-time data) is another matter all together.
2. Without Specification you are in trouble
You should always specify what you are doing, even if you are completely and utterly wrong !
This seems to be the most obvious statement of all time, however, time and time again I find software projects without specification. This then leads to a host of secondary problems; poor or no estimation; arguments or organisational confusion; lack of accountability; poor performance and others.
If you can't specify what you are doing, you can't estimate how long it will take and you therefore can't tell what resources you will need. Your standard project management triangle (scope, time, resource) is thus broken and all the vertices are variable. Immediately the project is doomed to either fail or underperform.
You have no accountability - without written definition, it is simply a `he/she said, he/she said match`. You also cannot iterate effectively, as there is no known starting point. You cannot negotiate, as nothing is defined and most importantly you cannot design !
The entire software development process starts from definition, even if this is for small chunks of the entire system. `Agile` methodologies have various levels of abstractions, Epics, Stories, Releases etc. There is a reason for this, most methodologies focus on how to write definitions and how to thereafter, estimate and allocate.
Without estimation, you cannot allocate resources and therefore you cannot produce roadmaps, this means that the business has no visibility of what it is doing, this is a recipe for disaster.
Further without specification, organisations tend to skip the architecture step and then the outcome may or may not converge on something robust and efficient. In fact the likelihood of producing not only bad software, but software that cannot be maintained is very high.
A good test to determine how well your project is defined, is to monitor how long the development team can go without leadership interfering or answering questions about features. If the product owner is constantly having to have meetings to explain things then it is probably not very well specified.
3. Interview for Excellence
In an environment where skilled resources are hard to find, it is tempting to be more flexible on the requirements in order to get `bums on seats`. This is a very dangerous game, engineering excellence needs the right team, with the right skill set and the right attitude, anything less and you are setting up the organisation for `engineering non-excellence`. The interview process should be taken very seriously and be very systematic.
Software is built by people, and it is very important to get the right people and the right teams in place. Always favour quality over quantity (unless there is some other hidden motivation).
4. Agile does not mean `do nothing`
As previously mentioned, `agile` is not a synonym for `do nothing` or `leave it till later`. Effective agile development requires a significant amount of planning, domain knowledge and `hypothesis testing` (from the Lean methodologies). Do not allow projects to drift without direction, be systematic in your approach.
5. `Devops` first is not always productive (working code first, devops after)
We live in a world of `Devops`, cloud services, where servers can be spun up from the outset to compensate for poor quality engineering. If you can get software working without these services in a reliable manner they will no doubt work well with these services. However the opposite is not true. We often spend far too much time using the latest and greatest managed service without focusing on the core functionality and robustness of the software we are building. When things go wrong, they will go horribly wrong and best case have a much higher than needed operational cost. Don't let `Devops` be an excuse for poor engineering.
For a description of some potential issues see `Release It! 2nd Edition pg 46`.
Buggy code can in some ways be compensated for by `autoscaling`, but beware of a large bill !
6. Too many resources is as bad, or worse than too little
`The mythical man month` springs to mind, throwing resources at a problem is not always the best way to solve a problem. Often this leads to poor results, frustration and other organisational issues that are very hard to solve down the line. If you can effectively solve a problem with careful thought and limited resource, you are sure that that the problem solution has been thought through and is effective. Simply throwing more resources at a problem, does not mean that you will solve the problem quicker. In fact it might slow you down.
- * It takes time for new people to bond with a team
- * It takes time for new people to understand complex problems and projects
- * Each new person adds a management overhead
- * Team which are too large are hard to manage
The phrase `too many cooks in the kitchen` also comes to mind, you don't want a lot of people and personalities justling with each other, this is unproductive.
7. Design before Develop, before Deploy
Before one jumps in and starts developing, there must be an element of design. It is fine to prototype in order to inform design, but the prototypes cannot become the thing that is deployed. Too often proof-of-concept's somehow become the real thing and land up in production. For obvious reasons, this is a bad idea.
It is always a good idea to mock the UI, and use this to help the UI designers figure out how they are going to develop the application. However, it is not a good idea to use the UI mocks as the specification for the product, especially since it misses out all the backend considerations and the non-functional requirements - which in complex systems are vitally important.
Comments
Post a Comment
Please post your comments, I am interested in your views