Blog‎ > ‎

IT Architecture


Immutable MicroServices

posted May 20, 2016, 6:15 PM by Jake Vosloo   [ updated May 20, 2016, 6:22 PM ]

I love the concept of immutable microservices. In short, it means that every component of a system is designed and packaged as a microservice and those components' configuration is never changed. This means, virtual servers, virtual switches, databases, applications, etc are all versioned and if a configuration change is required, a new copy of it is deployed on the correct version.

For this to work, the organisation has to understand a few base patterns:

Go live first IT methodology

posted Mar 11, 2016, 3:46 PM by Jake Vosloo   [ updated Mar 11, 2016, 3:50 PM ]

When developing a new system, companies often view go-live as the last step in the software development life cycle. This causes a mental disconnect in people’s minds having on the one hand the SDLC of: inception, design, implementation, installation and on the other hand the production support. In the modern time this mindset is no longer efficient enough, with companies moving towards a DevOps and NoOps operational strategy instead.

I’m proposing that regardless of which operational strategy companies use, they should also adopt a “Go live first” approach.  By this I mean that any project to develop a new system or replacement of an existing system should begin by installing a placeholder for the system into production and holding a go-live event for that placeholder. This means, the system has not been developed, the business requirements are not yet collected. All that is available is the high level vision of what the system will accomplish and a placeholder. During the "go-live first" event the management backers of the project explain the vision of the project to the future users of the project and show them how to access the placeholder and how to contact support about the system.

The business analysts then start writing business requirements and system architects write designs, but instead of writing it as if for a future system, they write it as changes to an existing system. The operational staff support the system as they do for any other production system.

The benefit of this is that it will stop companies from building development support systems that are not used to support production system. It will coordinate all IT development into focusing on the productionisation of the systems and how to support it while it is being changed. It will encourage IT staff to be more closely aligned with business and it will enable business people to think and act with more agility.

Software development as the translator in the man – machine communication model

posted Nov 6, 2013, 7:59 AM by Jake Vosloo

Software development is the process of enabling users to communicate instructions to a machine and have it perform those instructions as expected. More generically, software development is the action of building a communication channel between a user and a machine.

This communication between user and machine closely follows the Berlo SMCR model of communication as explained through the following example:

1.      Instruction

a.      Source – The user has an intention to have the machine perform an action.

b.      Message – The instruction which the user wish the machine to perform.

c.      Channel – The software accepts the message through a user interface and translates the message into the relevant format for the machine.

d.      Receiver – The machine receives the translated message and acts on it.

2.      Feedback

a.      Source – The machine has succeeded or failed to perform the instruction.

b.      Message – The feedback on the execution of the instruction.

c.      Channel – The software translates the feedback and presents it to the user through a user interface.

d.      Receiver – The user understands the feedback presented.

Note: Software systems may consist of a chain of channels, each being a receiver of a message and a source for the next link in the chain. As long as instructions originate from a user and gets executed by a machine, the above process holds for each link in the chain as well as the chain as a whole.

 

Software development is responsible for making this communication as effective as possible by limiting the translation failures (noise) at every step. To accomplish this, a software developer must optimize the following variables:

1.      Understanding of the user's intentions and expectations

2.      Understanding of the machines capabilities and language

3.      Translation of the message into the machine's language

 

An enterprise system has many thousands of users and many hundreds of machines combined into one software system.  This makes it near impossible for one person to optimize all 3 variables.

Software design principle - Robustness

posted Oct 22, 2011, 12:34 AM by Jake Vosloo   [ updated Oct 22, 2011, 1:24 AM ]

Robustness is probably one of the most important priciples in software development.  Even more important than Security or Reliability.
The reason is simple.  If the software is not robust then it can not be secure nor reliable.  Robustness indicates how well the system's other features perform their function and continue to perform their function.
Robustness is a principle and not something you do, if you copy what someone else does to create robust code, you misssed the plot.

Below are some articles on robustness, learn from their thinking process and not from their lists of things they do to improve robustness.
Alberto Gutierrez has a good set of 10 characteristics to learn from.
Matt Bishop's robust programming article.
You may draw a robustness diagram to see all the potential fragile points in your system.
Read more on how robustness is often overlooked or wrongly called "quality".
iMatix tells how they build reliable/robust systems.


Organization of the architect

posted Sep 26, 2011, 1:51 PM by Jake Vosloo   [ updated Sep 26, 2011, 1:58 PM ]

Based on the concept, Organization of the artist devised by architect Frank Gehry.  The organization of the artist places the architect/artist in control of the design throughout construction and deliberately eliminates the influence of politicians and business people on design.  Many IT projects go over budget or fail due to the architect not being in a position to truly run the project.  The architect must be mature enough to know the limit of his technology and development teams, he must also know how to get the true needs from the customer and not be swayed by the loud talkers and political players in the customer's ranks.

CouchBase Memcache & CouchDB NoSQL

posted Jun 30, 2011, 1:05 PM by Jake Vosloo


MongoDB, CouchDB, MySQL Compare Grid

CouchDB and jQuery Mobile

CouchDB guide

Part II. Developing with CouchDB

5. Design Documents

6. Finding Your Data with Views

7. Validation Functions

8. Show Functions

9. Transforming Views with List Functions

Part III. Example Application

10. Standalone Applications

11. Managing Design Documents

12. Storing Documents

13. Showing Documents in Custom Formats

14. Viewing Lists of Blog Posts


Couchbase HTTP API Reference

2. CouchDB API Server Database Methods
2.1. GET /db
2.2. PUT /db
2.3. DELETE /db
2.4. GET /db/_changes
2.5. POST /db/_compact
2.6. POST /db/_compact/design-doc
2.7. POST /db/_view_cleanup
2.8. POST /db/_ensure_full_commit
2.9. POST /db/_bulk_docs
2.9.1. Inserting Documents in Bulk
2.9.2. Updating Documents in Bulk
2.9.3. Bulk Documents Transaction Semantics
2.9.4. Bulk Document Validation and Conflict Errors
2.10. POST /db/_temp_view
2.11. POST /db/_purge
2.11.1. Updating Indexes
2.12. GET /db/_all_docs
2.13. POST /db/_all_docs
2.14. POST /db/_missing_revs
2.15. POST /db/_revs_diff
2.16. GET /db/_security
2.17. PUT /db/_security
2.18. GET /db/_revs_limit
2.19. PUT /db/_revs_limit
3. CouchDB API Server Document Methods
3.1. POST /db/doc
3.1.1. Specifying the Document ID
3.1.2. Batch Mode Writes
3.1.3. Including Attachments
3.2. GET /db/doc
3.2.1. Attachments
3.2.2. Getting a List of Revisions
3.2.3. Obtaining an Extended Revision History
3.2.4. Obtaining a Specific Revision
3.3. HEAD /db/doc
3.4. PUT /db/doc
3.4.1. Updating an Existing Document
3.5. DELETE /db/doc
3.6. COPY /db/doc
3.6.1. Copying a Document
3.6.2. Copying from a Specific Revision
3.6.3. Copying to an Existing Document
3.7. GET /db/doc/attachment
3.8. PUT /db/doc/attachment
3.8.1. Updating an Existing Attachment
3.9. DELETE /db/doc/attachment
4. CouchDB API Server Local (non-replicating) Document Methods
4.1. GET /db/_local/local-doc
4.2. PUT /db/_local/local-doc
4.3. DELETE /db/_local/local-doc
4.4. COPY /db/_local/local-doc
5. CouchDB API Server Design Document Methods
5.1. GET /db/_design/design-doc
5.2. PUT /db/_design/design-doc
5.3. DELETE /db/_design/design-doc
5.4. COPY /db/_design/design-doc
5.4.1. Copying a Design Document
5.4.2. Copying from a Specific Revision
5.4.3. Copying to an Existing Design Document
5.5. GET /db/_design/design-doc/attachment
5.6. PUT /db/_design/design-doc/attachment
5.7. DELETE /db/_design/design-doc/attachment
5.8. GET /db/_design/design-doc/_info
5.9. GET /db/_design/design-doc/_view/view-name
5.9.1. Querying Views and Indexes
5.9.2. Sorting Returned Rows
5.9.3. Specifying Start and End Values
5.9.4. Using Limits and Skipping Rows
5.9.5. View Reduction and Grouping
5.10. POST /db/_design/design-doc/_view/view-name
5.10.1. Multi-document Fetching
5.11. POST /db/_design/design-doc/_show/show-name
5.12. POST /db/_design/design-doc/_show/show-name/doc
5.13. GET /db/_design/design-doc/_list/list-name/other-design-doc/view-name
5.14. POST /db/_design/design-doc/_list/list-name/other-design-doc/view-name
5.15. GET /db/_design/design-doc/_list/list-name/view-name
5.16. POST /db/_design/design-doc/_list/list-name/view-name
5.17. PUT /db/_design/design-doc/_update/updatename/doc
5.18. POST /db/_design/design-doc/_update/updatename
5.19. ALL /db/_design/design-doc/_rewrite/rewrite-name/anything
6. CouchDB API Server Miscellaneous Methods
6.1. GET /
6.2. GET /_active_tasks
6.3. GET /_all_dbs
6.4. GET /_log
6.5. POST /_replicate
6.5.1. Replication Operation
6.5.2. Specifying the Source and Target Database
6.5.3. Single Replication
6.5.4. Continuous Replication
6.5.5. Canceling Continuous Replication
6.6. POST /_restart
6.7. GET /_stats
6.8. GET /_utils
6.9. GET /_uuids
6.10. GET /favicon.ico
7. CouchDB API Server Configuration Methods
7.1. GET /_config
7.2. GET /_config/section
7.3. GET /_config/section/key
7.4. PUT /_config/section/key
7.5. DELETE /_config/section/key


Brisk: Cassandra & Hadoop NoSQL mashup

posted Jun 24, 2011, 10:25 AM by Jake Vosloo   [ updated Jun 24, 2011, 10:49 AM ]

DataStax’ Brisk is an enhanced open-source Apache Hadoop and Hive distribution that utilizes Apache Cassandra for many of its core services. Brisk provides integrated Hadoop MapReduce, Hive and job and task tracking capabilities, while providing an HDFS-compatible storage layer powered by the Cassandra DB.
DataStax Brisk Apache Cassandra Hadoop

A key benefit of DataStax’ Brisk is the tight feedback loop it allows between real-time applications and the real time analytics that follow. Traditionally, users would be forced to move data between systems via complex ETL processes, or perform both functions on the same system with the risk of one impacting the other.

http://www.datastax.com/products/brisk

http://ria101.wordpress.com/2010/02/24/hbase-vs-cassandra-why-we-moved/

http://www.theregister.co.uk/2011/03/23/cassandra_mashed_with_hadoop/

http://allthingshadoop.com/2010/04/24/running-hadoop-mapreduce-with-cassandra-nosql/

http://pig.apache.org/

Rackspace casandra by example:
http://www.rackspace.com/cloud/blog/2010/05/12/cassandra-by-example/


Lean Software Development Plans for Change

posted Apr 5, 2011, 6:57 AM by Jaco Vosloo   [ updated Apr 5, 2011, 7:51 AM ]

The Kaizen way is to plan for change.  In software development this can be a bit of a problem because changing software is usually exponentially more expensive than it was to write in the first place.  To make software development lean, one must counteract this exponential change curve.  This is accomplished by using refactoring, automated testing and continuous integration in order of priority. Once this is in place, change and quality becomes cheap.

From my experience I cannot remember any projects where a heavy up front design has produced a working project, I can however remember many projects where the principles of refactoring has saved a failing project.  The most successfull projects used refactoring as a way of coding, from the start new features are refactored into existence.

For more information read Martin Fowler's excelent article on the same topic:
http://martinfowler.com/articles/designDead.html

CQRS Architecture

posted Feb 24, 2011, 12:24 AM by Jaco Vosloo   [ updated Feb 24, 2011, 12:30 AM ]

At last I found the architecture description for my preferred software design...

CQS and CQRS. This image by Udi Dahan explains it nicely.



Complex Event processing engine

posted Dec 2, 2010, 12:35 PM by Jaco Vosloo   [ updated Jun 24, 2011, 9:53 AM by Jake Vosloo ]

 CEP with DB4O

http://www.db4o.com/s/cepdb.aspx

http://developer.db4o.com/Solutions/ComplexEvents.aspx

Esper is an open-source CEP engine written entirely in Java and fully embeddable into any Java process - custom, JEE, ESB, BPM, etc. It recently has reached version 2.0, is backed by EsperTech under a professional open source / dual license business model, and triggers a growing interest in the Java and .Net community as well with NEsper, its full .Net/C# implementation. Both can be downloaded from esper.codehaus.org with complete examples and documentation.

The Esper engine works somewhat like a database turned upside-down. Instead of storing the data and running queries against stored data, Esper enables applications to store queries (or statements) and run the data through them. Response from the Esper engine is thus real-time when conditions occur that match queries. The execution model is thus continuous rather than only when a query is submitted.

One of the most attractive features in Esper is that is easy to formulate continuous queries: Esper provides an SQL-like language extended to handle event streams and patterns, an Event Processing Language (EPL). For developers more inclined to program against an API, a continuous query object model API is also available.

http://www.theserverside.com/news/1363826/Complex-Event-Processing-Made-Simple-Using-Esper

1-10 of 13